ResNet
Add a shortcut around every couple of layers, and suddenly a hundred-layer network trains — one line of arithmetic that unlocked depth.
When not to use it
- (There isn't a good case for omitting them in a deep network.)*
- In shallow networks. Under about ten layers there's nothing to rescue.
- Post-activation, when pre-activation exists. The follow-up paper is better and the original ordering is what most tutorials still show.
- Without projecting the shortcut when dimensions change. It won't add, and the error is confusing.
Reach for something else instead
- Dense connections (DenseNet) — concatenate rather than add. More parameters, similar motivation.
- Highway networks — gated shortcuts, predating ResNet. Gates turned out to be unnecessary.
- Careful initialisation (Fixup) — trains deep nets without normalization, and questions what residuals are for.
Sources & further reading
- He et al. (2015), Deep Residual Learning for Image Recognition — the paper; the degradation problem and the one-line fix.
- He et al. (2016), Identity Mappings in Deep Residual Networks — pre-activation; the clean gradient path, and the version you should use.
- Veit, Wilber & Belongie (2016), Residual Networks Behave Like Ensembles of Relatively Shallow Networks — the reframing that undercuts "deep."
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Building anything deep without them and blaming the depth. That's the solved problem.
- Using post-activation because the original paper did. The follow-up is better.
- Thinking residuals add capacity. They don't — the deep network could already represent the shallow one. They change what optimisation can find.
- Reading "150 layers" as 150 layers of processing. Veit et al. suggest it's an ensemble of shallow paths.
At a glance
FieldDeep Learning
The idea
y = F(x) + xWhy it worksthe gradient gets a
+1 path that can't vanishWhat it fixeddegradation: deeper nets with worse training error
Usepre-activation ordering
Reframingan ensemble of shallow networks, not one deep one
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
ResNet vs. plain deep network — the same layers, plus an addition. That addition is the difference between 20 trainable layers and 150.