Layer Normalization
Rescaling each example's activations so training stays stable — the normalization transformers actually use, and the reason batch normalization did not fit them.
Reviewed July 16, 2026Stable
When not to use it
- In convolutional vision models, where batch normalization generally still performs better.
- As a substitute for sensible initialisation rather than a complement to it.
Reach for something else instead
- Batch normalization for vision with large fixed batches.
- RMSNorm, a cheaper variant that omits mean-centring.
- Group normalization where batch sizes are small but spatial structure matters.
Sources & further reading
- Ba, Kiros & Hinton (2016), Layer Normalization — the original.
- Xiong et al. (2020), On Layer Normalization in the Transformer Architecture — pre-norm versus post-norm and why warmup was needed.
- Santurkar et al. (2018), How Does Batch Normalization Help Optimization? — challenges the standard explanation.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming transformers use batch norm. They do not, for reasons of batch and sequence independence.
- Treating pre-norm and post-norm as interchangeable. They differ in trainability at depth.
- Expecting normalization to rescue a badly initialised or badly scaled model on its own.
At a glance
FieldDeep Learning
Normalizes acrossfeatures of one example
Used bytransformers
Placementpre-norm dominant
Cheaper variantRMSNorm
DifficultyHands-on
Where this sits
5 concepts come first. Understanding it opens up 67 more.
5Levelsteps in
5Needs firstconcepts
67Opens up25% of 264
8Areascrosses fields
Computed from the prerequisite graph, not assigned. How this works