Benchmark
A standard test used to compare AI systems — indispensable for progress, and routinely mistaken for a measure of the thing it approximates.
Reviewed July 11, 2026Stable
When not to use it
- To decide whether a model works for your task. It can't tell you that. Thirty of your own examples can.
- When the benchmark is saturated. If everything scores 92%, the differences are noise dressed as signal.
- As evidence of intelligence, understanding, or reasoning. It's evidence of performance on the benchmark. The rest is inference and it's contested.
- When the benchmark predates the model by years. Contamination is likely and the score is closer to a memorisation test.
Reach for something else instead
- Your own evaluation set — thirty real examples. The most valuable artefact in any AI project.
- Executable benchmarks (SWE-bench style) — objective, harder to game, closer to real work.
- Human preference evaluation — catches what automated tests miss; brings its own confounds.
- A/B testing in production — the only measure of whether users got what they needed.
This entry is part of a longer guide: What is AGI?
Read more on the blog
- AI in science: 380,000 predicted, 736 actually madeAlphaFold won a Nobel Prize and did not reduce the rate of experimental structure determination. A materials model predicted 380,000 stable compounds and 736 have been synthesised.
- How to check an AI claim before you believe itA vendor advertised a hallucination rate below 0.001% with no evidence behind it, and a state attorney general investigated. Six questions separate a claim you can act on from a number someone typed.
- Fixing the seed does not make it reproducibleTraining the same network fifty times with an identical seed produced almost as much variance as fifty different seeds. Roughly 80% of the spread came from GPU arithmetic, not from randomness anyone controls.
- What a model card should say and usually does notA model card was meant to be an evidentiary document: what a model was trained on, where it fails, how it was measured. Regulation has now made most of it mandatory, and the gap between the template and its honest use is the whole subject.
Further reading
- Chollet (2019), On the Measure of Intelligence — benchmarks measure skill, not intelligence; the case for efficiency of acquisition instead.
- Hendrycks et al. (2021), Measuring Massive Multitask Language Understanding — MMLU, and worth reading for what its authors claim it measures versus how it gets cited.
- Sainz et al. (2023), NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for each Benchmark — the contamination problem stated plainly.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading a benchmark score as a general capability claim. It's a claim about one test.
- Comparing scores across differently-configured runs. Prompt format, few-shot count and parsing all move numbers by several points.
- Ignoring the saturation point — celebrating 94% vs. 92% on a benchmark where both are noise.
- Assuming decontamination worked. It's best-effort substring matching against an unauditable corpus.
- Building your product around a leaderboard rank rather than your own thirty examples.
At a glance
FieldFoundations
What it measuresperformance on itself
What it's read asgeneral capability
Main threatscontamination, saturation, Goodhart
Best alternativethirty of your own examples
DifficultyBeginner
Often compared with
Benchmark vs. your own eval set — one tells you the model is in the right league; the other tells you whether it does your job. Only one of those is a decision.
Where this sits
5 concepts come first. Understanding it opens up 4 more.
3Levelsteps in
5Needs firstconcepts
4Opens up1% of 310
3Areascrosses fields
Computed from the prerequisite graph, not assigned. How this works