How do we measure AI progress? The benchmark problem

Every new AI model arrives with a table of numbers: this many percent on one benchmark, that many on another, a green arrow showing it beat the last model. The numbers look like objective proof of progress, the AI equivalent of a lab result, and they drive billions of dollars in decisions and most of the public sense of how fast the field is moving. But a growing body of careful work says something uncomfortable: those numbers are far less trustworthy than they look. Benchmark scores are not self-interpreting, because the tests saturate, leak into the training data, and reward optimization to the test rather than the ability they were meant to measure, which means our capacity to measure AI has quietly fallen behind our capacity to build it.

This is the evaluation problem, and it has become one of the hardest and least visible parts of AI. This guide explains why evaluation matters so much, how AI systems are actually measured, the four ways benchmark numbers mislead, saturation, contamination, gaming, and measuring the wrong thing, the newer methods built to fix them, and the honest state of a field that can now build systems faster than it can rigorously test them. The goal is not to make you distrust every number, but to let you read a benchmark score for what it is: useful evidence that needs interpretation, not a verdict.

Why evaluation matters more than it seems

Evaluation is the instrument the whole field reads its progress on, and that makes it foundational in a way that is easy to overlook. Every claim that one model is better than another, every report that AI crossed some threshold, every decision about whether a system is safe to deploy, rests on a measurement. Benchmarks are how researchers compare approaches, how companies decide what to ship, how buyers choose between products, and how the public forms a picture of how capable these systems are. If the instrument is distorted, everything downstream inherits the distortion: real progress becomes hard to distinguish from the appearance of progress, regressions slip through, and resources flow toward whatever scores well rather than whatever works. A broken thermometer does not just give a wrong reading; it corrupts every decision made on the basis of the reading. That is why the quiet difficulty of measuring AI matters as much as the loud advances in building it.

How AI is actually measured

There are three main ways models are evaluated, each with a different trade-off. The most familiar is the benchmark: a fixed set of test questions with known correct answers, covering a skill like general knowledge, grade-school or competition mathematics, coding, or graduate-level science. A model's answers are scored automatically against the answer key, producing a single comparable number, which is why benchmarks power leaderboards and launch announcements. When benchmarks fall short, teams turn to two alternatives. Human preference arenas pit two models against each other on open-ended prompts and let people vote on which answer is better, aggregating the votes into a rating much like a chess ranking. And LLM-as-a-judge uses one AI model to grade another's outputs, which is fast and scalable enough to evaluate things no fixed answer key can capture. Each method measures something real, and each, as the rest of this piece shows, introduces its own distortions.

Problem one: saturation

The first way benchmarks mislead is that they wear out. As models improve, driven by the scaling laws that reliably raise capability, they push scores on a fixed test toward the ceiling, and once the best models all cluster near the top, around ninety percent and above, the differences between them stop being meaningful, drowned in noise and the test's own imperfections. Many of the benchmarks still quoted in comparisons crossed this point years ago; some widely cited knowledge and coding tests saturated as early as 2023 and 2024. Worse, the pace is accelerating: benchmarks deliberately designed to be hard for the best models of their year are now often near-solved within a couple of years of release, so the field burns through its yardsticks faster and faster and has to keep commissioning harder ones, up to recent expert-written exams built specifically to be difficult for frontier systems.

It is worth being precise here, because saturation is partly a success story. A benchmark reaching the ceiling can mean the field truly mastered the capability it measured, which is good news. The problem is not that benchmarks saturate but that saturated benchmarks keep being cited as if they still separate the best models, when in fact they no longer can. A ninety-something score on a saturated test tells you a model has joined the leading pack, not that it leads it, and reading small gaps on such a test as real differences is one of the most common ways benchmark numbers deceive.

Problem two: contamination

The second problem is more insidious. Benchmarks are usually public so that everyone can use them, which means their questions and answers sit on the open web, and the open web is exactly what models are trained on. So a model may have seen the test, or close variants of it, during training. When that happens, its score no longer measures whether it can solve those problems; it partly measures whether it memorized them, and memorization masquerading as capability is precisely what a test is supposed to rule out. This is called contamination or data leakage, and it is a form of overfitting to the test set that classical machine learning has warned about for decades, now operating at the scale of the entire internet.

The most convincing demonstration came from a simple experiment. Researchers hand-wrote a brand-new grade-school math test in the exact style and difficulty of a famous existing benchmark, then ran the leading models on both. If a model had truly learned arithmetic, the two scores should match. For several model families they did not: scores dropped by as much as thirteen points on the fresh test, and the size of each model's drop tracked how often it would spontaneously reproduce the original benchmark's problems word for word. The models had learned the test better than they had learned the skill. Audits of other popular benchmarks have since found training-data overlap across frontier models as a rule rather than an exception. Contamination is very hard to detect after the fact, which is why an old, public benchmark should be treated as contaminated until proven otherwise.

Problem three: Goodhart's law

The third problem is structural, and it has a name from economics: Goodhart's law, the principle that when a measure becomes a target, it ceases to be a good measure. A benchmark works as a proxy for some real ability only as long as no one is optimizing directly for the benchmark. The moment a score becomes the thing everyone competes on, the thing that sells models and wins headlines, effort flows into raising that specific number, whether by training on similar data, tuning to the benchmark's format, or subtler forms of teaching to the test. As that happens, the score and the underlying ability it was meant to represent drift apart, because it is almost always easier to raise the measured proxy than the real thing it stands for. The benchmark keeps going up while the capability it once tracked no longer moves with it, and a tool built to measure progress quietly turns into a tool for marketing it. This is not usually outright cheating; it is the ordinary consequence of optimizing hard against any fixed target, the same dynamic that makes reward signals drift from intended goals throughout AI.

Problem four: measuring the wrong thing, and the judge problem

The fourth problem is the deepest, because it questions whether the number means what we think even when nothing is broken. A benchmark measures a narrow, specific proxy, and the leap from that proxy to real-world usefulness is often large. A model that aces a coding benchmark is not therefore a good software engineer, because real engineering involves understanding messy requirements, working across a large codebase, and judgment that a self-contained puzzle does not test. This gap between what a benchmark measures and what a capability actually requires is a construct validity problem, and studies putting models into real deployment have found large gaps between benchmark performance and how well the same models do on genuine tasks. High benchmark scores can coexist with disappointing real-world behavior, and often do.

When static benchmarks saturate or leak, teams reach for the two alternatives named earlier, and both move the measurement onto shakier ground. Human preference arenas ask which answer people prefer, which is not the same as which is correct: people tend to prefer answers that are longer, more confident, and more agreeable, so a model can climb the rankings by being persuasive and flattering rather than right, a bias toward sounding good that overlaps with sycophancy. LLM-as-a-judge inherits the judging model's own biases, favoring certain answer positions, longer responses, and, revealingly, outputs that resemble its own, and it cannot reliably check correctness in specialized fields like medicine or law where being wrong is most costly. There is a subtler danger too: unlike a multiple-choice test, whose errors are more or less random, a judge model's errors are correlated with the thing being measured, so they do not average out but systematically reshape the rankings. And because judge models are themselves updated over time, evaluation scores can shift even when the model being evaluated has not changed at all.

What good evaluation looks like now

None of this is a reason to stop evaluating; it is a reason to evaluate more carefully, and the field is responding with better methods. The most direct answer to contamination is freshness: benchmarks that draw only on material created after a model's training cutoff, continuously harvesting new problems from recent competitions, papers, and news, so the questions cannot have been seen in training; this matters especially for reasoning models, whose multi-step problem-solving is easy to fake through memorization and hard to measure honestly. Scores on these live benchmarks are markedly lower and more spread out than on the saturated classics, which is exactly what an honest measurement should look like. A second answer is privacy: keeping the test set secret and accessible only through a controlled interface, so it cannot be trained on directly, trading some transparency for validity. A third is designing benchmarks to stay hard, as with recent expert-authored exams whose questions were pre-screened against frontier models and discarded if the models could already answer them. Beyond fixing individual benchmarks, the emerging consensus is that no single number suffices, and serious evaluation is layered: automated benchmarks for broad coverage, model-based judging for scalable screening, and human experts for the correctness that matters most, plus perturbed or held-out variants to expose contamination and real-task trials to check that lab scores survive contact with reality. The through-line is the same idea that makes synthetic data work: measurement is only as good as the verifier behind it, and building trustworthy verifiers is now central work.

The honest state of things

The accurate summary is that benchmark scores remain genuine evidence but are not self-interpreting, and reading them well requires holding four questions in mind: has this benchmark saturated, could it be contaminated, is it being optimized as a target, and does it actually measure the ability in question. The deeper situation is that measurement has fallen behind capability. The field can now build models faster than it can rigorously evaluate them, which is why so much frontier work is really evaluation work, and why the launch numbers that dominate the conversation are systematically more flattering than the reality behind them. This is not a minor technical annoyance. It bears directly on safety and alignment, because a system's safety-relevant properties are exactly the ones we most need to measure and least know how to, and scores on a safety benchmark can rise while the underlying property stays unverified, in the same way a model's hallucination rate can look fine on a test yet fail in the wild. It bears on trust, since buyers and users are routinely guided by numbers that overstate what they will experience. And it bears on progress itself, because optimizing a broken measure means optimizing noise. Understanding AI's real trajectory means learning to read its benchmarks with the same care you would bring to any other statistic that someone has a strong incentive to make look good.

The short version

AI progress is measured mainly through benchmarks, fixed sets of test questions with known answers that produce comparable scores, supplemented by human preference arenas and by using one model to judge another. All three are less reliable than they appear. Benchmarks saturate, with the best models clustering near the ceiling so that differences stop being meaningful, and the field burns through them ever faster. They suffer contamination, because public test questions leak into web-scraped training data, so scores partly measure memorization, dramatically demonstrated when models scored far lower on a freshly written version of a famous math test. They fall to Goodhart's law, since once a score becomes the target everyone optimizes, it drifts away from the ability it was meant to track. And they face construct-validity problems, measuring narrow proxies that can diverge sharply from real-world usefulness, while the fallback methods, preference arenas and model judges, carry their own systematic biases. Better approaches, using fresh post-cutoff questions, private held-out sets, deliberately hard expert-written exams, and layered human-and-automated evaluation, are emerging in response.

The idea to hold onto is that a benchmark score is useful evidence, not a verdict, because saturation, contamination, Goodhart's law, and weak construct validity all push the headline numbers to look better than the underlying capability, which means the ability to measure AI has fallen behind the ability to build it, and rigorous evaluation has quietly become one of the hardest problems in the field. Read the numbers the way you would read any statistic that someone is strongly motivated to inflate.

Common questions

How is AI evaluated? AI is evaluated mainly through benchmarks, which are fixed sets of test questions with known correct answers covering skills like knowledge, mathematics, coding, or science; a model's answers are scored automatically to produce a comparable number. When benchmarks fall short, two other methods are used: human preference arenas, where people vote on which of two models gave the better answer to open-ended prompts, and LLM-as-a-judge, where one AI model grades another's outputs at scale. Each method measures something real but introduces its own distortions, which is why interpreting evaluation results carefully matters as much as running them.

What is an AI benchmark? An AI benchmark is a standardized test used to measure and compare model capabilities. It consists of a fixed set of tasks or questions with known correct answers, in an area such as general knowledge, grade-school or competition mathematics, coding, or graduate-level science. A model is run on the questions and scored against the answer key, producing a single number that can be compared across models and over time. Benchmarks power leaderboards and launch announcements because they make progress look objective and quantifiable, but their scores need careful interpretation, since saturation, contamination, and optimization pressure can all make them misleading.

Why are AI benchmarks unreliable? Benchmark numbers mislead in four main ways. They saturate: as models improve, scores cluster near the ceiling and stop distinguishing the best systems. They suffer contamination, because public test questions leak into web-scraped training data, so a high score can reflect memorization rather than skill. They fall to Goodhart's law, since once a benchmark becomes the target everyone optimizes for, the score drifts away from the ability it was meant to measure. And they have construct-validity limits, measuring narrow proxies that can differ sharply from real-world usefulness. The fallback methods, human preference arenas and model judges, add their own biases, so no single number should be taken at face value.

What is benchmark saturation? Benchmark saturation is when the best models all score near the top of a test, around ninety percent and above, so the differences between them become statistically meaningless and the benchmark can no longer tell them apart. Many widely cited benchmarks saturated years ago, and the pace is accelerating, with tests designed to be hard often near-solved within a couple of years. Saturation is partly a success, since it can mean the field mastered the measured skill, but the problem is that saturated benchmarks keep being cited as if they still separate the leading models, when a top score now means a model has joined the leading pack rather than leads it.

What is data contamination in AI evaluation? Data contamination, or leakage, is when a model has seen benchmark questions or closely related content during training, because public test sets end up in the web data models are trained on. This inflates scores, since the model partly recalls answers rather than working them out, so the benchmark measures memorization instead of capability. It was shown clearly when researchers wrote a fresh version of a famous math test in the same style: several models scored much lower on the new version, and the drop tracked how often each model reproduced the original questions verbatim. Contamination is hard to detect after the fact, so old public benchmarks should be treated as potentially contaminated.

What is Goodhart's law in the context of AI? Goodhart's law states that when a measure becomes a target, it ceases to be a good measure. In AI, a benchmark works as a proxy for a real ability only while no one is optimizing directly for it. Once a score becomes the thing models compete on and sell on, effort flows into raising that specific number through training on similar data, tuning to the format, or teaching to the test, and the score drifts away from the underlying ability, because raising the measured proxy is usually easier than improving the real thing. The benchmark keeps climbing while the capability it once tracked does not, turning a measurement tool into a marketing one.

How should AI be evaluated properly? Good evaluation combines several defenses. Use fresh benchmarks whose questions come only from material created after a model's training cutoff, so they cannot have been memorized, or private held-out test sets that are never made public. Design tests to stay truly hard, for example expert-written exams pre-screened so that questions the best models already answer are discarded. And rather than trusting one number, evaluate in layers: automated benchmarks for broad coverage, model-based judging for scalable screening, and human experts for the correctness that matters most, plus real-task trials to confirm that lab scores hold up in deployment. The guiding principle is that a measurement is only as trustworthy as the verifier behind it.