How to Read AI Model Benchmarks Without Buying Marketing

Conversations about switching models increasingly start with a printed leaderboard. Someone points at a table where the new model beats the incumbent by a few pp and asks whether that justifies a migration. An honest answer requires stepping back and asking where those pp came from.

Benchmarks make sense as a research instrument. The trouble starts when a lab result lands on a sales slide and begins to pose as a forecast of how the model will handle a particular company’s invoices, contracts or service tickets. Three mechanisms account for most of the gap between the two.

Contamination: the answer key leaked

Benchmarks get published on the internet. Training data gets collected from the internet. The rest of the story writes itself: the longer a test set circulates in public, the higher the chance a model has already seen the problems, or their solutions, during training.

Scale AI put a number on this in May 2024. The team built GSM1k, 1,205 fresh grade school math problems¹ matched to the widely used GSM8K set on difficulty, number of reasoning steps and problem length. The most overfitted model families lost up to low double digits in pp, while frontier models dropped far less. The conclusion is measured: an overfitted model can still solve novel problems, it simply performs worse than its public score promised.

A similar story played out on SWE-bench Verified (software engineering benchmark), a set of 500 issues lifted straight from open source repositories that served for two years as the unofficial scoreboard for coding ability. In February 2026 OpenAI published its reasoning for no longer reporting results on that set: growing contamination combined with flaws in the tests themselves meant that differences at the top of the table stopped carrying information. It pointed instead to SWE-bench Pro, released by Scale AI in September 2025, where part of the task pool is deliberately held back from publication so that overfitting can be detected. At launch, the strongest models scored under 25% there.²

For a buyer, the creation date of a test set matters as much as the score: a 2021 set, where frontier models have long been scraping the ceiling of the scale, mostly measures memorization.

Benchmarks measure a different skill

MMLU (Massive Multitask Language Understanding), for years the most quoted general knowledge test set, is roughly fourteen thousand multiple choice questions across 57 subjects. In 2024 a research team manually reviewed a sample of 5,700 of those questions³ and published a corrected version called MMLU-Redux. Their estimate is that around 6.5% of items contain errors³, ranging from a wrong answer key to ambiguous questions and items with several defensible answers. Some subjects were far worse, with more than half the reviewed virology questions found faulty. Filtering out the broken items was enough to reorder models in the rankings.

Even with a perfect answer key, a heavier difference remains. Corporate work rarely arrives with four options to choose from. It looks more like twenty pages scanned at 2013 quality, industry jargon, internal abbreviations, a requirement to cite the source of every number, and consequences that only show up in the books weeks later. The ability to pick the right item from a list correlates with that work only partially.

Trap	How it shows	What to do before buying
Test set contamination	The score rises, the work does not	Check when the set was created
Errors in the answer key	Rankings shift once corrected	Ask which version of the set was scored
A different skill than the job needs	Multiple choice versus a 20-page scan	Build your own set from company documents
Closed task pool	No way to detect overfitting	Prefer sets with a held-out portion

Variance: same model, different numbers

The third mechanism tends to be the most underrated. A benchmark score describes the whole harness around the model as much as the model itself: system prompt, available tools, number of attempts, verification method, token budget. On agentic tasks that layer accounts for a very large share of the spread between published numbers, and it is often summarized in a single footnote.

Then there is the question of who reported the result. Most entries on public leaderboards come from the vendors themselves and have never been independently reproduced. In April 2025 researchers from Cohere Labs, AI2 and several universities published an analysis titled “The Leaderboard Illusion”, describing among other things the practice of testing many private model variants before release and showing only the best of them. Ahead of the Llama 4 launch the figure reported was 27 variants. The arena operators accepted some of the criticism and disputed other parts, but the selection effect is a problem the reader of a ranking is left holding.

Models also change under a fixed name. A version update or a shift in default parameters can move behaviour on edge cases, which means last quarter’s table describes a slightly different object.

Your own test set beats leaderboards

The alternative is less impressive looking and considerably more useful. We build it like this:

• Where cases come from: thirty to sixty drawn from production work, including the hard and edge cases where the process usually stalls. Clean cases mainly measure the optimism of whoever assembled the set.
• Who defines the output: the person who does this work daily, not the technical team. Without that, evaluation degenerates into an argument about taste.
• When criteria get written: before anyone sees results. The order matters, because criteria added afterwards reliably confirm whatever conclusion was already preferred.
• The same harness: identical prompt, identical tools, identical number of attempts for every candidate. Otherwise the comparison is between configurations rather than models.
• What we measure alongside quality: cost and response time. A model two points better and four times more expensive rarely wins on an annual view.
• Where the set lives: with us only. Publishing it to a public repository starts exactly the contamination cycle that spoiled the industry benchmarks.

Building a set like this usually takes one to three days of work from someone who knows the domain. The compromise is obvious: fifty cases is a small sample, so differences of a few pp on it mean just as little as they do on a public board. What the set does reliably distinguish is whether a model copes with our documents or fails on them, and that is precisely the information a leaderboard cannot supply.

Four questions before quoting

When a table makes it into a meeting anyway, run it through a short filter. Who reported the score and whether anyone independent reproduced it. When the task set was created and whether it could already sit in training data. Whether a held out pool exists so that overfitting can be detected. How the harness is described: prompt, tools, number of attempts.

A leaderboard is good enough to narrow a shortlist from twenty candidates to three. The decision about which of those three goes into production rests on our own data.

Footnotes

1. Scale AI, “A Careful Examination of Large Language Model Performance on Grade School Arithmetic”, May 2024: https://arxiv.org/abs/2405.00332. ↩

2. Scale AI, SWE-Bench Pro, September 2025: https://scale.com/blog/swe-bench-pro. ↩

3. Gema et al., “Are We Done with MMLU?”, 5,700-question sample, estimated 6.49% error rate: https://arxiv.org/abs/2406.04127. ↩ ↩²