Reproducibility
Getting the same result twice, which turns out to be several different claims of very different strength, and which GPU arithmetic defeats even when the seed is fixed.
Reviewed July 16, 2026Stable
When not to use it
- As a proxy for correctness. A result can be bitwise reproducible and fail to generalise.
- As a demand at frontier scale, where repeating the run is not available at any price outside the lab.
Reach for something else instead
- Repeatability with reported variance, appropriate where runs are expensive but possible.
- Artifact availability plus external evaluation, where the training cannot be repeated.
- Version pinning and distributional reporting, where the model is behind someone else's API.
Read more on the blog
- Testing a system that answers differently every timeForty percent of organisations hit significant quality regressions within ninety days of deploying an LLM application. Exact-match assertions are the cause: they reject valid answers and occasionally accept wrong ones.
- AI in science: 380,000 predicted, 736 actually madeAlphaFold won a Nobel Prize and did not reduce the rate of experimental structure determination. A materials model predicted 380,000 stable compounds and 736 have been synthesised.
- How to check an AI claim before you believe itA vendor advertised a hallucination rate below 0.001% with no evidence behind it, and a state attorney general investigated. Six questions separate a claim you can act on from a number someone typed.
- AI in software: 19% slower, and they felt 20% fasterA randomised trial put experienced developers 19% behind on their own repositories while they reported being 20% ahead. The researchers had expected a speedup and published the opposite.
Further reading
- Pham et al. (2020), Problems and Opportunities in Training Deep Learning Software Systems — same-seed variance from non-determinism.
- Goldblum et al. (2020), Trained Model Reproducibility — the taxonomy of reproducibility claims.
- Pineau et al. (2021), Improving Reproducibility in Machine Learning Research — the checklist adopted at major venues.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Believing a fixed seed is sufficient, when most of the variance is elsewhere.
- Conflating reproducibility with replicability, which are different claims.
- Chasing bitwise identity, which can select for brittleness: a result holding only under one deterministic configuration is weaker than one holding across many.
At a glance
FieldApplied AI
Same-seed variance~74% of different-seed
Causenon-associative float addition on parallel hardware
Cheapest fixcontainerisation
Six tiersrepeatability to conceptual replicability
DifficultyPractical
Where this sits
A destination. 6 concepts lead here, and nothing in the corpus depends on it.
4Levelsteps in
6Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstBenchmark
Computed from the prerequisite graph, not assigned. How this works