Artifipedia
GlossaryMapBlog
Home/Deep Learning/Batch Normalization
Deep Learning

Batch Normalization

Renormalising activations at every layer — one of deep learning's most important techniques, and its original explanation turned out to be wrong.

Reading level: Curious
Pick your depth ↓

When not to use it

  • In transformers. LayerNorm or RMSNorm. Batch statistics across variable-length sequences don't mean anything.
  • With small batches. Statistics from 2 examples are noise. Use GroupNorm.
  • With dropout, carelessly. The variance shift between them can make the pair worse than either.
  • When inference must not depend on the batch. BatchNorm couples examples; that's occasionally unacceptable.

Reach for something else instead

  • LayerNorm — per-example, no batch dependence. Transformers.
  • RMSNorm — LayerNorm without mean subtraction. Cheaper, standard in current LLMs.
  • GroupNorm — small-batch vision.
  • No normalization — with careful initialisation and residual scaling. Credible, and it questions the whole practice.

Sources & further reading

  • Ioffe & Szegedy (2015), Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — the paper. Note the title contains the explanation that didn't survive.
  • Santurkar et al. (2018), How Does Batch Normalization Help Optimization? — the refutation; it smooths the loss landscape, and covariate shift isn't the mechanism.
  • Ba, Kiros & Hinton (2016), Layer Normalization — the batch-independent version that transformers use.

Primary sources, listed so you can check the claims on this page rather than take them on trust.

Where people go wrong

  • Repeating "it reduces internal covariate shift." That explanation was tested and didn't hold.
  • Using it with a batch of 2 and wondering why training is unstable.
  • Forgetting model.eval(), so inference uses batch statistics and depends on what else was in the batch.
  • Leaving bias=True on a layer followed by BatchNorm. The bias is subtracted away.

At a glance

FieldDeep Learning
What it doesrenormalise activations per batch, then learn a scale and shift
Original explanationinternal covariate shift, disproven
Actual mechanismsmooths the loss landscape (contested)
Use in transformersno; LayerNorm/RMSNorm
Breaks onsmall batches
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4

Often compared with

BatchNorm vs. LayerNorm — one normalises across the batch and couples your examples; the other normalises per example. Transformers needed the second.

Related concepts

Deep Learning

Neural Network

A system of simple connected units that learns patterns from examples — the foundation underneath deep learning and modern AI.

Deep Learning

Batch Size

How many examples the model sees before each update — a systems constraint that everyone treats as a hyperparameter.

Deep Learning

Vanishing Gradient

The signal dying on its way back through a deep network — the problem that kept deep learning impossible for twenty years.

Foundations

Deep Learning

Machine learning using neural networks with many layers — the approach behind nearly every recent AI breakthrough.

Deep Learning

Dropout

Randomly switching off neurons during training — the technique that defined an era of deep learning and has quietly disappeared from modern architectures.

Deep Learning

Transformer

The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.