Fixing the seed does not make it reproducible

Train a standard image network fifty times with different random seeds and record the spread in test accuracy. Then train it fifty more times with the same seed, changing nothing at all, and record that spread.

The second number should be zero. It is not. In a controlled study of exactly this, the same-seed standard deviation was about 74% of the different-seed standard deviation.

Roughly 80% of the run-to-run variance came from sources that fixing the seed does not touch.

The previous article in this series argued for running multiple seeds before believing a result. That advice stands and it is incomplete, because the seed is not where most of the variance lives.

Reproducibility in machine learning is usually discussed as a discipline problem, solved by releasing code and fixing seeds. A substantial part of it is an arithmetic problem that no amount of discipline addresses, and knowing which part you are facing determines whether the fix is a checklist or a rebuild.

Why identical code gives different answers

The cause is specific and it is not a bug in anyone's software.

Floating-point addition is not associative. With finite precision and rounding, (a + b) + c and a + (b + c) can differ. The standard demonstration: adding 1 twice to a very large number gives one result if you add them one at a time, because each addition rounds away, and a different result if you add the two ones together first, because their sum survives the rounding.

On a GPU, a large tensor operation is split across thousands of threads and the partial results are combined. The order in which those partial sums combine depends on thread scheduling, which depends on hardware state nobody controls. Two runs of identical code sum the same values in a different order and produce slightly different numbers.

Then those differences pass through nonlinearities, where they are amplified rather than averaged away, and propagate through every subsequent layer. A discrepancy in the last decimal place of an early activation becomes a different gradient, a different weight update, and eventually a different model.

This is why the same-seed variance was three quarters of the different-seed variance. Seeds control initialisation, data order and dropout. They do not control the order in which a GPU adds numbers.

What fixing seeds actually buys

Worth being precise, because the advice to fix seeds is correct and oversold.

Seeds work when execution is not parallel. For CPU training, or for the non-GPU parts of a pipeline, fixing the seed produces identical results. This is a real property and it covers a smaller share of modern work every year.

Seeds are necessary and not sufficient. Major frameworks default to nondeterministic algorithms built on atomic operations, chosen for speed. Setting a seed does not change the algorithm selection, so the nondeterminism remains regardless of how carefully the seed is managed.

Seeds are plural. Python's generator, the numerical library's, the framework's CPU generator, the framework's GPU generator, the data loader's per-worker generators, and a hash-randomisation environment variable are all separate. Fixing one is common; fixing all of them is a specific piece of work.

And seeds do not travel. Even fully deterministic settings are not guaranteed to give bitwise identical results across different GPU architectures or different counts of streaming multiprocessors. A result reproducible on your machine may not be on someone else's, which is exactly the case that matters for anyone checking your work.

The hierarchy nobody distinguishes

Discussions of this get confused because one word covers several claims of very different strength. The literature separates them.

Repeatability. The same team, same code, same hardware, gets the same result. The weakest claim and the one most often meant when someone says "reproducible".

Computational reproducibility. Bitwise identical results from the same code, same data, same software environment and same hardware setup. This is what containerisation targets and what GPU nondeterminism defeats.

Dependent reproducibility. A different team gets the same result using the original artifacts. This is what a code release enables.

Independent reproducibility. A different team gets the same result building from the description alone. Much stronger and much rarer.

Direct replicability. A different team, running a new experiment of the same design, reaches the same conclusion.

Conceptual replicability. The finding holds under a different design testing the same hypothesis. The strongest claim and the one science actually wants.

Almost all of the reform effort in machine learning targets the first three, because they are checkable. The claim that matters is the last one, and nothing in a reproducibility checklist establishes it. A result can be bitwise reproducible and still not generalise, which is the failure that costs anyone anything.

What actually works

The interventions with evidence behind them, roughly in order of return.

Containerisation. Studies of reproduction attempts find that shipping the full software environment materially increases success rates. Most failed reproductions are not subtle scientific disagreements; they are a library version that changed behaviour, a dependency that was removed, or an operating system difference. A container removes an entire class of failure and is the cheapest item on this list.

Deterministic algorithm modes. Frameworks expose settings that force deterministic kernel selection and disable autotuning benchmarks that pick different algorithms per run. Setting these plus every seed produces genuine determinism on fixed hardware, at a performance cost.

Patching. A systematic approach that identifies nondeterministic operations and substitutes deterministic equivalents, combined with recording and replaying execution. Applied carefully, this reproduced several open-source models and one commercial model exactly. It works and it is real engineering rather than a configuration flag.

Reporting the environment. Hardware, driver versions, library versions, and whether deterministic mode was enabled. This is nearly free and routinely omitted, which means a reader cannot tell whether a discrepancy is scientific or numerical.

What determinism costs

The trade is real and is usually left out of advocacy.

Speed. Deterministic kernels are slower than the ones chosen for throughput. On large training runs this is a direct increase in the largest cost in the project.

Output quality, for generation. Deterministic inference generally means sampling at temperature zero, and zero temperature frequently produces worse output than a small positive value. Enforcing reproducibility on a generative system therefore degrades the thing being reproduced.

And provider APIs do not offer it. One major provider's documentation states plainly that determinism is not guaranteed and that users should expect almost always the same result. Anyone building on a hosted model has no path to bitwise reproducibility regardless of local discipline.

That last point matters more than it appears. A growing share of published results uses hosted models, so the reproducibility conversation is arguing about a property those results structurally cannot have.

The scale problem

Everything above assumes the training can be run again.

For frontier systems it cannot. Reproducibility is constrained by the sheer cost of retraining, and a result costing millions of dollars per run is not going to be independently reproduced by anyone. The training data is frequently undisclosed, the compute is unavailable, and the exact configuration is proprietary.

This is not a discipline failure and it is not fixable by a checklist. It is a structural condition, and the field has not settled what evidentiary standard replaces reproduction when reproduction is impossible.

The partial answers in circulation: evaluate the artifact rather than the process, since the model can be tested even if the training cannot be repeated; require external evaluation on held-out material the lab did not construct; and treat single-lab results as provisional in a way the current publishing culture does not.

None of these is established practice.

What standard is achievable

The useful conclusion is a set of tiers rather than a single demand, because demanding bitwise reproducibility of a frontier training run is not a serious position.

For small and mid-scale work: computational reproducibility is achievable and should be expected. Container, all seeds, deterministic modes, environment reported. The cost is a performance penalty and an afternoon of setup.

For large training runs: repeatability with reported variance. Not bitwise identical, but run several times with the spread reported, and the environment documented well enough that a discrepancy can be attributed.

For frontier results: artifact availability and independent evaluation. The training cannot be reproduced, so the claim to be checked is about the model rather than the process, and the check has to be run by someone other than the lab.

For anything built on a hosted API: state the model version and date, accept that bitwise reproduction is unavailable, and report across enough runs that the conclusion does not depend on any one of them.

Which tier your work is in, and what to actually do

The tiers above are only useful if you can place yourself, so here is the placement test and the resulting obligation.

Can you rerun the training on hardware you control, in under a day, for under a few hundred pounds? You are in the reproducible tier. Containerise, seed everything, enable deterministic modes, report the environment. There is no defensible reason not to, and the whole setup is an afternoon.

Can you rerun it, but only on shared or rented hardware, taking days and real money? Repeatability with reported variance. Run three times, report the spread, document the environment thoroughly enough that someone attributing a discrepancy can tell whether it is numerical or scientific. Bitwise identity is not worth the performance penalty at this scale.

Can you not rerun it at all, because it cost more than your annual budget? Artifact availability. The claim someone else can check is about the model, not the process, so publish weights or provide access, and expect the meaningful verification to be external evaluation on material you did not construct.

Are you calling someone else's API? Version pinning and distributional reporting. Record the model identifier and date, run each evaluation enough times to characterise the spread, and state plainly that bitwise reproduction is unavailable. A result that depends on one API response is not a result.

The reason to place yourself explicitly is that most methodological argument in this area is people in different tiers talking past each other. Someone demanding containerised bitwise reproduction of a frontier training run and someone dismissing reproducibility because their setting makes it impossible are both applying a standard from the wrong tier, and neither is wrong within their own.

What is unresolved

Whether determinism is worth its cost at scale. Deterministic training is slower and the variance it removes may be smaller than the variance from other sources. Nobody has established the point at which the performance penalty stops being worth the guarantee.

What replaces reproduction for frontier work. The field is publishing results that cannot be independently reproduced and has not agreed what evidence should be required instead. This is the largest open question in research methodology for the field and it is being deferred.

Whether hardware nondeterminism affects conclusions or only numbers. Small numerical differences produce different models. Whether those models differ in ways that change what a paper concludes, or only in which specific examples they get wrong, is not resolved, and the answer differs by setting.

How much unreproducibility is unreproducible findings. A failed reproduction can mean the result was wrong, the description was incomplete, or the environment differed. These are very different diagnoses and reproduction studies frequently cannot distinguish them.

The counter-argument

Bitwise reproducibility may be the wrong target. What matters scientifically is whether the conclusion holds, not whether the twelfth decimal place matches. A field that achieved perfect computational reproducibility and never checked whether findings replicate would have optimised the wrong thing, and there is a version of this debate that mistakes precision for reliability.

The variance finding cuts both ways. If 80% of run-to-run variance is hardware nondeterminism rather than seed randomness, that is an argument that seed variance is a smaller problem than assumed, not only that hardware is a larger one. The total spread was modest in absolute terms.

Determinism can hide fragility. A result that only holds under one deterministic configuration is more brittle than one that holds across many nondeterministic runs. Chasing bitwise reproduction could select for exactly the wrong property.

And the cost falls unevenly. Requiring containerisation, deterministic modes and full environment reporting is a modest burden for a well-resourced lab and a real one for a small group. Standards that raise the floor also raise the barrier.

The short version

Training a standard image network fifty times with an identical seed produced a standard deviation about 74% as large as fifty runs with different seeds, meaning roughly 80% of run-to-run variance came from sources that fixing the seed does not touch.

The cause is that floating-point addition is not associative. On a GPU, tensor operations are split across thousands of threads and the order in which partial sums combine depends on thread scheduling, so identical code sums identical values in a different order. The resulting differences pass through nonlinearities where they amplify rather than cancel, and propagate into different gradients, different weights and eventually a different model.

Fixing seeds is necessary and not sufficient. It works where execution is not parallel, frameworks default to nondeterministic algorithms regardless of seeding, seeds are plural across at least six generators, and even correct settings are not guaranteed identical across GPU architectures.

One word covers claims of very different strength: repeatability, computational reproducibility, dependent and independent reproducibility, direct and conceptual replicability. Reform effort targets the first three because they are checkable, and the claim that matters is the last, which no checklist establishes. A result can be bitwise reproducible and fail to generalise.

What works: containerisation, which removes an entire class of environment failures and is the cheapest intervention; deterministic algorithm modes plus complete seeding; systematic patching of nondeterministic operations, which has reproduced real models exactly; and reporting the environment. What it costs: speed, and for generation, output quality, since deterministic inference means temperature zero. Hosted APIs do not offer determinism at all, and a growing share of published work uses them.

And for frontier systems none of this applies, because the training cannot be run again at any price outside the lab that ran it. The field has not agreed what evidence replaces reproduction when reproduction is impossible, which is the largest open methodological question it has and the one being deferred.

Common questions

Why do I get different results with the same random seed? Because seeds do not control everything. GPU tensor operations split work across thousands of threads, and floating-point addition is not associative, so the order in which partial results combine changes the answer in the last decimal places. That order depends on thread scheduling rather than on anything you set. In a controlled study, same-seed variance was about 74% of different-seed variance, meaning roughly 80% of the spread was not seed-related.

How do I make machine learning training reproducible? Four things together. Containerise the full software environment, which removes the largest class of failures. Fix every seed, which means the language runtime, numerical library, framework CPU and GPU generators, data loader workers and hash randomisation. Enable deterministic algorithm modes and disable autotuning benchmarks that select different kernels per run. And report the hardware, driver and library versions so a discrepancy can be attributed. Expect a performance cost.

Is fixing the random seed enough for reproducibility? No. Seeds produce identical results when execution is not parallel, and major frameworks default to nondeterministic algorithms built on atomic operations regardless of seeding. Even with correct settings, results are not guaranteed bitwise identical across different GPU architectures or different streaming multiprocessor counts, so a result reproducible on your machine may not be on the machine of whoever checks your work.

What is the difference between reproducibility and replicability? Reproducibility usually means obtaining the same result from the same artifacts, which subdivides into repeatability by the same team, computational reproducibility as bitwise identity, and dependent or independent reproduction by others. Replicability means reaching the same conclusion from a new experiment, either of the same design or a different one testing the same hypothesis. Reform effort targets reproducibility because it is checkable, while replicability is what science actually wants and no checklist establishes it.

Why does floating-point arithmetic cause this? Because addition with finite precision is not associative: rounding means the grouping of operations changes the result. Adding one twice to a very large number gives a different answer depending on whether you add them separately, where each rounds away, or together, where their sum survives. GPUs compute large operations in parallel, so grouping depends on scheduling, and the resulting differences amplify passing through nonlinear functions rather than averaging out.

Can I get reproducible results from a hosted model API? Not bitwise. At least one major provider states explicitly that determinism is not guaranteed and that users should expect almost always the same result. Seed parameters where offered reduce variation without eliminating it. The practical response is to state the model version and date, run enough times that your conclusion does not depend on any single response, and report the spread rather than a single output.

What does deterministic training cost? Speed, primarily, since deterministic kernels are slower than the throughput-optimised ones frameworks select by default, and on large runs that is a direct increase in the dominant project cost. For generative systems it also costs output quality, because deterministic inference generally means sampling at temperature zero, which frequently produces worse results than a small positive temperature.

How can frontier model results be verified if they cannot be reproduced? There is no settled answer, which is the largest open methodological question in the field. Training runs costing millions cannot be repeated by anyone outside the lab, the data is frequently undisclosed, and the configuration is proprietary. The partial proposals in circulation are to evaluate the artifact rather than the process, require external evaluation on held-out material the lab did not construct, and treat single-lab results as provisional. None is established practice.