Residual Connection
Adding a layer's input to its output so gradients have an unobstructed path backwards — the single change that made networks deeper than about twenty layers trainable at all.
Reviewed July 16, 2026Stable
When not to use it
- In shallow networks, where there is no gradient path long enough to need one.
- As a substitute for sensible initialisation rather than a complement to it.
Reach for something else instead
- Dense connections, where each layer receives every earlier layer's output.
- Highway networks, the gated predecessor that learned how much to pass through.
- Careful initialisation schemes that train deep networks without shortcuts.
Sources & further reading
- He et al. (2016), Deep Residual Learning for Image Recognition — the original, and the degradation problem it solved.
- Veit, Wilber & Belongie (2016), Residual Networks Behave Like Ensembles of Relatively Shallow Networks — the ensemble interpretation.
- Xiong et al. (2020), On Layer Normalization in the Transformer Architecture — why pre-norm keeps the residual path clear and post-norm does not.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Putting a normalization or transformation on the shortcut, which obstructs the path the connection exists to keep clear.
- Treating pre-norm and post-norm as interchangeable. They differ in trainability at depth.
- Assuming it only helps gradients. The ensemble effect suggests it changes what is computed, not only how it trains.
At a glance
FieldDeep Learning
Computesinput plus F(input)
Solvesdegradation with depth
Enablesnetworks past ~20 layers
Placement matterspre-norm keeps the path clear
DifficultyHands-on
Where this sits
A destination. 10 concepts lead here, and nothing in the corpus depends on it.
8Levelsteps in
10Needs firstconcepts
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works