Batch Normalization
Renormalising activations at every layer — one of deep learning's most important techniques, and its original explanation turned out to be wrong.
When not to use it
- In transformers. LayerNorm or RMSNorm. Batch statistics across variable-length sequences don't mean anything.
- With small batches. Statistics from 2 examples are noise. Use GroupNorm.
- With dropout, carelessly. The variance shift between them can make the pair worse than either.
- When inference must not depend on the batch. BatchNorm couples examples; that's occasionally unacceptable.
Reach for something else instead
- LayerNorm — per-example, no batch dependence. Transformers.
- RMSNorm — LayerNorm without mean subtraction. Cheaper, standard in current LLMs.
- GroupNorm — small-batch vision.
- No normalization — with careful initialisation and residual scaling. Credible, and it questions the whole practice.
Batch norm re-centres each layer's outputs to keep training stable.
As data flows through a deep network, the distribution of each layer's inputs drifts, which slows and destabilises training. Batch normalization rescales each layer's outputs to a consistent mean and variance across the batch, keeping the signal in a healthy range. The result is faster, more stable training and less sensitivity to initialisation.
The full account
The explanation in the title
Ioffe and Szegedy's 2015 paper is one of the most cited in deep learning, and its title states a causal claim: Accelerating Deep Network Training by Reducing Internal Covariate Shift. The argument was intuitive and it spread instantly. As a network trains, each layer's weights change, so the distribution of inputs arriving at the next layer keeps shifting underneath it. Every layer is chasing a moving target. Normalise each layer's inputs to zero mean and unit variance, the story goes, and you stop the target moving. Training gets faster.
The technique worked spectacularly. It let people train much deeper networks, tolerate higher learning rates, and care less about initialisation. Within two years it was in essentially every architecture. And because the technique worked, nobody seriously interrogated the explanation attached to it.
That is the whole problem, and it is a general one. A mechanism that works is taken as evidence for the story told about why it works. Those are different claims, and only the first one had support.
The refutation
Santurkar, Tsipras, Ilyas and Madry tested the explanation directly in 2018, and their paper's subtitle is not a summary but a verdict: (No, It Is Not About Internal Covariate Shift).
The decisive experiment is elegant enough to describe in a sentence. If BatchNorm works by reducing internal covariate shift, then deliberately increasing internal covariate shift should break it. So they injected random noise — with a distribution changing at every time step — into the activations immediately after the BatchNorm layer. This produces severe, artificial covariate shift: the layer downstream now receives inputs whose distribution is unstable by construction, exactly the condition the theory says BatchNorm exists to prevent.
The networks trained fine. Noisy BatchNorm networks performed comparably to clean ones and still substantially better than networks without BatchNorm at all. The proposed mechanism was removed and the benefit remained. In the other direction, they found no consistent relationship between how much a layer's input distribution moved and how well the network trained — networks with more covariate shift sometimes trained better.
Their positive account is that BatchNorm reparameterises the optimisation problem so that the loss landscape is smoother — technically, it improves the Lipschitz constants of both the loss and its gradients. A smoother landscape means the gradient at your current point remains a good predictor of the gradient a step away, which is exactly the property that lets you take larger steps safely. That explains the higher learning rates directly, without reference to distributions at all.
Why this one matters beyond BatchNorm
Lipton and Steinhardt made the sharpest version of the point in the same year, in Troubling Trends in Machine Learning Scholarship, where BatchNorm is the worked example of a specific failure: explanation offered as though it were finding. The internal covariate shift story was speculation in the original paper. It was repeated as mechanism by the field. And the repetition, not the evidence, is what made it true.
Three years passed between the claim and the test. During those three years, internal covariate shift was taught in courses, cited in hundreds of papers, and used to motivate new normalisation techniques designed to reduce a quantity that turned out not to be the operative one. The technique kept working the entire time, which is precisely why nobody checked.
| The 2015 story | What 2018 found | |
|---|---|---|
| Mechanism | Stabilises the distribution of layer inputs | Smooths the loss landscape (better Lipschitz constants) |
| Test | None reported | Add noise to increase ICS — BatchNorm still works |
| ICS vs. performance | Assumed inverse | No consistent relationship observed |
| Explains high learning rates? | Indirectly, by hand-waving | Directly — a predictive gradient permits larger steps |
What is still open
It would be neat to end with "so it's the smoothing." It is not that settled. Kohler and colleagues published a competing account in 2018 — that BatchNorm's benefit comes from decoupling the length and direction of the weight vectors, turning the optimisation into a problem gradient descent handles better. Others have argued the smoothing result depends on assumptions that don't hold in general, and that the regularisation effect of batch statistics — each example's normalisation depends on the other examples in the batch, which injects noise — carries part of the benefit.
So the honest position in 2026 is that BatchNorm is a decade old, universally deployed, empirically indispensable, and nobody has an agreed account of why it works. The wrong explanation is refuted; the right one is contested.
The part that should bother you
Search for a batch normalisation tutorial today and you will find internal covariate shift, presented as the mechanism, with no mention of Santurkar. Not in archived material — in pages written in the last year. The refutation is seven years old, was published at NeurIPS, and is titled so unambiguously that you cannot cite the paper without stating its conclusion.
That is the useful thing this page has to tell you, and it is not about normalisation. A field can carry a refuted explanation for a decade with no cost, because the technique still works and the explanation was never load-bearing for anything except understanding. If you want to know whether an explanation in machine learning has been tested, the default assumption should be that it has not.
Further reading
- Ioffe & Szegedy (2015), Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — the paper. Note the title contains the explanation that didn't survive.
- Santurkar et al. (2018), How Does Batch Normalization Help Optimization? — the refutation; it smooths the loss landscape, and covariate shift isn't the mechanism.
- Ba, Kiros & Hinton (2016), Layer Normalization — the batch-independent version that transformers use.
- Ioffe & Szegedy (2015), Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — the original, and the explanation in its own title is the part that didn't survive.
- Santurkar, Tsipras, Ilyas & Madry (2018), How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) — injected noise to deliberately increase ICS; BatchNorm kept working.
- Lipton & Steinhardt (2018), Troubling Trends in Machine Learning Scholarship — uses BatchNorm as the case study in explanation offered without evidence.
- Kohler et al. (2018), Exponential Convergence Rates for Batch Normalization — the length-direction decoupling account, an alternative to the smoothing story.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Repeating "it reduces internal covariate shift." That explanation was tested and didn't hold.
- Using it with a batch of 2 and wondering why training is unstable.
- Forgetting
model.eval(), so inference uses batch statistics and depends on what else was in the batch. - Leaving
bias=Trueon a layer followed by BatchNorm. The bias is subtracted away.
At a glance
Often compared with
Where this sits
A destination. 5 concepts lead here, and nothing in the corpus depends on it.
Computed from the prerequisite graph, not assigned. How this works