1How often the frontier disagrees On 67% of claims (672 / 1,000; 95% CI: 64–70%), the frontier panel doesn’t agree — at least one model dissents from the majority verdict, or no strict majority forms at all. The breakdown: For each claim we looked at the five frontier verdicts and asked: did at least three pick the same answer (a strict majority)? If yes, how many of the remaining models dissented? If no clear majority emerged at all — verdicts split across three or four different buckets — the claim falls in the Models split, no majority row. Most of these claims are unlikely to appear in any training corpus with a gold label attached — there’s no canonical answer key to pattern-match against, no benchmark leaderboard to anchor to. We refer below to the “majority” and…
Five frontier LLMs disagree on 67% of 1k real-world fact-check claims
Advertisement