Benchmark Contamination
When the test is in the training data — the problem that makes most published model scores impossible to fully trust.
Reviewed July 12, 2026Stable
When not to use it
- (It's a hazard, not a technique. The question is when to assume it.)*
- Always, on any public benchmark. The prior should be that contamination is possible, not that it's absent.
- Especially on benchmarks older than the model. Years of discussion, paraphrase and posting.
- Especially on closed models. The decontamination claim is unfalsifiable from outside.
- Never on your own internal data. Nobody trained on your tickets. That's the value.
Reach for something else instead
- (Ways to get an uncontaminated measurement.)*
- Your own thirty examples — from your use case. The only test set nobody has seen.
- Post-cutoff problems — created after the model's training data ends.
- Private held-out sets — effective, and unverifiable from outside.
- Executable tasks — where the answer is computed rather than recalled.
Read more on the blog
- Why AI benchmarks mislead: contamination, gaming, saturationEvery model launch leads with benchmark scores, and buyers read them like thermometer readings. They are closer to opinion polls: directionally useful, methodology-dependent, and easy to game. Here is why the numbers mislead, from training-data contamination to Goodhart's law to saturation, and how to read them without being fooled.
- How do we measure AI progress? The benchmark problemEvery AI model launches with a table of benchmark scores that look like objective proof of progress. A growing body of evidence says those numbers are far less trustworthy than they appear, because benchmarks saturate, leak into training data, and reward the wrong thing. This is the evaluation problem, and it has quietly become one of the hardest parts of AI.
Further reading
- Sainz et al. (2023), NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for each Benchmark — the problem stated plainly.
- Golchin & Surdeanu (2023), Time Travel in LLMs: Tracing Data Contamination in Large Language Models — detecting it from outside, without corpus access.
- Zhou et al. (2023), Don't Make Your LLM an Evaluation Benchmark Cheater — how contamination inflates scores, and what it does to comparisons.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading a decontamination claim as a guarantee. It's substring matching against a corpus nobody can fully audit.
- Ignoring indirect contamination. The benchmark file isn't in the corpus; the blog post solving it is.
- Comparing a new model to an old benchmark and treating the gap as progress.
- Missing iterative contamination — a field tuning against a public benchmark for years has fitted to it collectively, and no procedure fixes that.
At a glance
FieldFoundations
The problemthe test set is in the training data
Why it's hard to detectmemorised and reasoned answers look identical
Detection from outsidepost-cutoff problems, guided prompting
Indirect formdiscussion of the benchmark, not the benchmark
Unfixable forma field tuning against it for years
DifficultyIntermediate
Often compared with
Benchmark contamination vs. overfitting — one is the model memorising the test set; the other is the model memorising the training set. The first is worse because it's invisible in the number.
Where this sits
A destination. 6 concepts lead here, and nothing in the corpus depends on it.
4Levelsteps in
6Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstBenchmark
Computed from the prerequisite graph, not assigned. How this works