Reproducibility
Getting the same result twice, which turns out to be several different claims of very different strength, and which GPU arithmetic defeats even when the seed is fixed.
Reviewed July 16, 2026Stable
When not to use it
- As a proxy for correctness. A result can be bitwise reproducible and fail to generalise.
- As a demand at frontier scale, where repeating the run is not available at any price outside the lab.
Reach for something else instead
- Repeatability with reported variance, appropriate where runs are expensive but possible.
- Artifact availability plus external evaluation, where the training cannot be repeated.
- Version pinning and distributional reporting, where the model is behind someone else's API.
Sources & further reading
- Pham et al. (2020), Problems and Opportunities in Training Deep Learning Software Systems — same-seed variance from non-determinism.
- Goldblum et al. (2020), Trained Model Reproducibility — the taxonomy of reproducibility claims.
- Pineau et al. (2021), Improving Reproducibility in Machine Learning Research — the checklist adopted at major venues.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Believing a fixed seed is sufficient, when most of the variance is elsewhere.
- Conflating reproducibility with replicability, which are different claims.
- Chasing bitwise identity, which can select for brittleness: a result holding only under one deterministic configuration is weaker than one holding across many.
At a glance
FieldApplied AI
Same-seed variance~74% of different-seed
Causenon-associative float addition on parallel hardware
Cheapest fixcontainerisation
Six tiersrepeatability to conceptual replicability
DifficultyPractical
Where this sits
A destination. 6 concepts lead here, and nothing in the corpus depends on it.
4Levelsteps in
6Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstBenchmark
Computed from the prerequisite graph, not assigned. How this works