Why machine learning does not do error bars

Train the same model on the same data with the same hyperparameters, changing only the random seed, and measure two things.

Accuracy varies by 0.057%. Effectively nothing.

Certified robustness varies by 28.9%.

Same models. Same training. The difference is which property you measured, and the second figure is larger than the improvements reported in papers claiming progress on it.

The standard complaint about machine learning is that it does not report error bars. The more useful observation is that variance is wildly unevenly distributed: negligible for headline accuracy on standard benchmarks, and large enough to swamp reported gains for derived metrics, shifted distributions and small datasets. Demanding error bars everywhere misses the point. Knowing where they matter is the actual skill.

The variance is smaller than critics assume

Start with the finding that complicates the standard critique, because taking it seriously changes what to argue for.

Careful measurement of training variance on standard image benchmarks found that the run-to-run standard deviation in distribution-wise error rate is around 0.03%. Different seeds produce networks that are, on the underlying distribution, essentially equally good.

There is a subtlety that explains why this feels wrong. Test-set variance is noticeably larger than test-distribution variance. Different seeds produce networks that make errors on different specific examples, so a fixed test set will rank them differently. But those errors are approximately independent, and the underlying competence is nearly identical.

The practical consequence is uncomfortable for a certain style of criticism: for standard training on standard benchmarks, a single run is more defensible than it looks. The seed lottery affects which examples get missed, not how good the model is.

If a paper reports one run of a well-established training recipe on ImageNet, the missing error bar is a real omission and probably not a large one.

Where it concentrates

The variance did not disappear. It moved, and it moved to places nobody routinely checks.

Derived and downstream metrics. The headline case. Certified robustness, verified through formal analysis, showed a standard deviation of 28.9% among models whose accuracy varied by 0.057%. A small difference in a decision boundary propagates into a large difference in what can be formally proven about it. Any metric computed through another process rather than measured directly can amplify variance this way, and calibration, fairness measures and interpretability scores are all candidates.

Shifted distributions. On the training distribution, seeds agree. On a shifted evaluation set, the same study found standard deviation roughly eight times higher. Since out-of-distribution performance is what most deployments care about, this is the variance that matters and the one least often reported.

Small datasets and short training. With one epoch of training, varying all randomness sources produced a standard deviation of 1.33%, against the 0.03% seen at full training length. Fine-tuning runs, low-resource settings and anything with limited data sit closer to the first figure.

Reinforcement learning and agent evaluation. Where the system interacts with a stochastic environment, variance compounds rather than averaging out. This is the setting where single-run reporting is least defensible and, historically, most common.

The rule that falls out: variance is small where the field has converged on a recipe and large everywhere else. Which means it is smallest exactly where papers report it and largest exactly where they do not.

The problem that no amount of variance reporting solves

There is a deeper issue, and it survives every methodological fix.

Comparisons of single scores necessarily yield positive results if experiments are repeated often enough. Run enough configurations, and some will beat the baseline by chance. Report the one that did, and you have a paper. Nothing in the reported experiment is false.

This is the multiple comparisons problem, well understood in every field that has confronted it, and machine learning has confronted it less than most. The search over architectures, hyperparameters, learning rates and seeds is a search over hypotheses, and the reported result is the maximum of that search rather than a draw from it.

Error bars on the final run do not address this. The variance being reported is the variance of the winner, conditional on having won.

The fixes are known and rarely applied: report the search budget alongside the result, so a reader knows how many configurations produced the winner; use random rather than grid hyperparameter search, which has been shown to shield against contradictory conclusions; and report performance as a function of compute rather than as a single point, since a method that wins at one budget may lose at another.

The counterintuitive finding

One result deserves separate mention because it inverts the natural intuition and changes what a good protocol looks like.

The instinct when controlling an experiment is to hold everything fixed except the thing under test. Fix the initialisation, fix the data order, fix the augmentation, vary only the seed you care about.

Varying as many sources of randomness as possible between runs actually decreases the variance of the true performance estimate.

The reason is that you are trying to estimate expected performance over the distribution of things that vary in practice. Holding sources fixed gives you a precise estimate of a configuration nobody will reproduce, since their initialisation and data order will differ. Varying everything gives a noisier individual measurement and a better estimate of what the method actually delivers.

A related finding sharpens it: at full training length, varying any single source of randomness produces about as much variance as varying all three. The sources are not additive, so controlling two of them buys much less than it appears to.

Why the field does not do this

The usual explanation is carelessness. The real reasons are more defensible and worth stating fairly.

Compute cost. Running a frontier training five times to report a standard deviation multiplies the largest expense in the project by five. For work at that scale, error bars are not a methodological choice but a budget decision, and the honest response is to acknowledge that the result is a single sample rather than to pretend otherwise.

The comparison would be uninformative anyway. Where variance is 0.03% and the claimed improvement is two points, the error bar tells the reader something they could have assumed. Effort spent producing it is effort not spent on an ablation, which is more informative.

Statistical machinery does not fit the setting well. Standard significance testing assumes independent samples and a null hypothesis. Multiple training runs of the same configuration are not independent in the relevant sense, the distributions are not normal, and the quantity of interest is usually an effect size rather than a rejection. Tooling has appeared to address this and adoption is limited.

And the incentives point elsewhere. A paper reporting that its improvement is within noise does not get accepted. This is a structural problem and no individual researcher fixes it.

Structural change has come from venues rather than individuals. Major conferences now require an explicit statement about error bars, including which sources of variability they capture, how they were computed, and whether they represent standard deviation or standard error. That is a meaningful shift and it is compliance rather than culture, which is what venue requirements can achieve.

What to do about it

As a reader, ask where the variance would be. Not whether error bars are present, but whether this is a setting where variance is small or large. A single run of a standard recipe on a standard benchmark: probably fine. A single run reporting a derived metric, an out-of-distribution result, a fine-tune on a small dataset, or anything involving an environment: treat the number as one draw.

Compare the effect size to the plausible spread, which is the same discipline as reading a paper's baselines. If the claimed improvement is smaller than the standard deviation you would expect in that setting, the result is a hypothesis. This requires knowing typical variance for your setting, which is worth establishing once for the tasks you work on.

As a practitioner, run your own three seeds before believing your own result. This is the cheapest and most-skipped check available. Most internal claims of improvement evaporate under it, and finding out privately is preferable.

Vary everything, not one thing. Different initialisation, different data order, different augmentation seed. It produces a noisier individual number and a better estimate of what you will actually get.

Report the budget. How many configurations were tried before this one. That single number does more for a reader's calibration than an error bar on the winner.

A variance table you can calibrate against

The advice to compare an effect size against plausible spread requires knowing the spread, which is the part nobody supplies. These are order-of-magnitude anchors drawn from the measurement literature, not values for your setting.

Two things to take from this rather than the specific numbers.

The spread across settings is three orders of magnitude. Any general rule about how many seeds to run is wrong for most of the table. The question is always which row you are in.

The rows where variance is largest are the rows where reporting is weakest. In-distribution image classification is the most-reported and least-variable setting. Interactive agents are the least-reported and most-variable. That inversion is not a coincidence: the settings with high variance are also the settings that are expensive to repeat, and the same constraint that produces the variance suppresses its measurement.

The practical move is to establish the figure once for the kind of work you do, by running one configuration five times and recording the spread. That number then serves as your threshold for every subsequent result, and it costs one afternoon rather than being repeated per experiment.

What is unresolved

Whether variance can be predicted rather than measured. If typical variance for a training setup could be estimated in advance, practitioners could assess significance without repeated runs. Work exists deriving such estimates for specific settings and no general method does, which would be the single most useful development here.

What the right statistical framework is. Borrowing null-hypothesis testing from other fields fits awkwardly, since the assumptions do not hold and the question is usually about effect size rather than rejection. Whether machine learning needs its own framework, or should adopt an existing one properly, is unsettled.

Whether frontier results can be assessed at all. The recommendations above assume you can run the training more than once. For results costing millions per run, nobody outside the lab can, and no established norm covers what confidence to assign a single expensive sample.

How much reported progress is noise. The uncomfortable question. If a substantial share of published improvements fall inside the variance for their setting, then aggregate progress is smaller than the literature suggests. Nobody has audited this systematically and the finding that variance in some derived metrics exceeds typical reported gains is not encouraging.

The counter-argument

Variance is small in the most common setting, and an article emphasising it risks encouraging dismissal of results that are fine. A standard deviation of 0.03% means most reported improvements on standard benchmarks are real, and reflexive scepticism about single-run results is miscalibrated.

Error bars can be theatre. A paper reporting mean and standard deviation over three seeds has satisfied a checklist and not established much: three samples give a poor variance estimate, and the number can be produced without changing any conclusion. Compliance is not rigour.

The compute argument is real, not an excuse. Demanding five runs of every experiment would substantially reduce the volume of work done, and it is not obvious that the field would learn more from a fifth as many results reported five times.

And other fields did not solve this either. Medicine and psychology have error bars, pre-registration and significance testing, and both had replication crises anyway. Statistical machinery is neither necessary nor sufficient for reliable findings, and importing it wholesale may import the failure modes with it.

The short version

Models differing only by random seed showed 0.057% standard deviation in accuracy and 28.9% in certified robustness, a figure larger than the improvements reported in papers claiming progress on that metric. Variance in machine learning is not uniformly absent from reporting; it is unevenly distributed, and the distribution is the thing to know.

For standard training on standard benchmarks it is small. Careful measurement found run-to-run standard deviation in distribution-wise error around 0.03%, meaning different seeds produce networks of nearly identical underlying competence. Test-set variance appears larger because seeds miss different specific examples, and those errors are approximately independent. A single run of a well-established recipe is more defensible than it looks.

The variance concentrates elsewhere: in derived metrics computed through another process, which can amplify a tiny difference in decision boundary into a large difference in what is provable; on shifted distributions, where the same study found roughly eight times more variance and which is what deployments care about; in short training and small datasets, where one epoch produced 1.33% against 0.03% at full length; and in any setting with a stochastic environment. Variance is smallest where the field has converged on a recipe and largest everywhere else, which means smallest where it gets reported.

A deeper problem survives every fix: comparisons of single scores necessarily yield positive results if enough configurations are tried, so the reported number is the maximum of a search rather than a draw from it. Error bars on the winner describe the variance of the winner conditional on winning. The fixes are reporting the search budget, using random rather than grid search, and reporting performance against compute.

And the finding that inverts intuition: varying as many sources of randomness as possible between runs decreases the variance of the true performance estimate, because holding sources fixed gives a precise estimate of a configuration nobody will reproduce. At full training length, varying one source produces about as much variance as varying all three.

Common questions

Do machine learning papers report error bars? Inconsistently, and the situation is more nuanced than the complaint suggests. Major venues now require an explicit statement about whether error bars are present, what sources of variability they capture, how they were computed, and whether they are standard deviation or standard error. Compliance has improved. Whether the reported bars are informative depends heavily on the setting, since variance in machine learning is very unevenly distributed.

How much does random seed affect neural network results? Much less than commonly assumed for standard training on standard benchmarks, where run-to-run standard deviation in distribution-wise error is around 0.03%. Different seeds produce networks with nearly identical underlying competence, though they miss different specific examples, which makes test-set variance look larger than true performance variance. Seed effects are far larger for derived metrics, shifted distributions, short training runs and small datasets.

Why is certified robustness so much more variable than accuracy? Because it is a derived metric. Certified robustness is computed through a formal verification process rather than measured directly, so a small difference in where a decision boundary sits can propagate into a large difference in what can be proven about it. Models with 0.057% standard deviation in accuracy showed 28.9% in certified robustness. Any metric computed through another process can amplify variance this way, including calibration and fairness measures.

How many random seeds should I run? At least three before believing your own result, which is the cheapest and most frequently skipped check available, and most internal claims of improvement do not survive it. More matters where variance is large: out-of-distribution evaluation, small datasets, short training, derived metrics, or anything involving a stochastic environment. Where you are running a well-established recipe on a standard benchmark, additional seeds buy relatively little.

Should I hold everything fixed except the seed I am testing? No, and this inverts the usual intuition. Varying as many sources of randomness as possible between runs decreases the variance of the true performance estimate, because you are estimating expected performance over the things that actually vary in practice. Holding initialisation and data order fixed gives a precise estimate of a configuration nobody else will reproduce. At full training length, varying one source produces about as much variance as varying all three.

Why do error bars not solve the significance problem? Because comparisons of single scores necessarily produce positive results if enough configurations are tried. The reported result is the maximum of a search over architectures, hyperparameters and seeds, not a draw from a distribution, so an error bar on the final run describes the variance of the winner conditional on having won. The relevant fixes are reporting the search budget, using random rather than grid hyperparameter search, and reporting performance as a function of compute.

Is it acceptable to report a single training run? It depends entirely on the setting and it is sometimes fine. For a standard recipe on a standard benchmark where variance is around 0.03%, a single run tells you nearly as much as five. For frontier-scale training costing millions per run, repetition is a budget decision rather than a methodological choice, and the honest response is to state that the result is one sample. For derived metrics, shifted distributions, small datasets or stochastic environments, a single run should be read as one draw.

How do I judge whether an improvement is real? Compare the effect size against the plausible spread for that setting rather than asking whether error bars are present. This requires knowing typical variance for the kind of work you do, which is worth establishing once. If the claimed improvement is smaller than the standard deviation you would expect, the result is a hypothesis rather than a finding, however many decimal places it carries.