Vanishing Gradient
The signal dying on its way back through a deep network — the problem that kept deep learning impossible for twenty years.
When not to use it
- (It's a failure mode, not a technique. The equivalent is when to suspect it.)*
- When early layers barely move. Log gradient norms per layer; the answer will be visible.
- In RNNs over long sequences. This is the original case and it's what LSTMs were built for.
- Whenever you see sigmoid or tanh in hidden layers. Derivative caps at 0.25. It's arithmetic.
- In a deep network without residuals. There's no reason to build one in 2026.
Reach for something else instead
- (Fixes, not substitutes.)*
- Residual connections — the structural answer. Gradient gets an unattenuated path.
- ReLU-family activations — gradient of 1, no attenuation.
- He / Xavier initialisation — start with the variance preserved.
- Gradient clipping — for the exploding version. Cap the norm.
Sources & further reading
- Hochreiter (1991), Untersuchungen zu dynamischen neuronalen Netzen — the thesis that identified the problem, years before anyone could act on it.
- Glorot & Bengio (2010), Understanding the Difficulty of Training Deep Feedforward Neural Networks — Xavier initialisation, and a clear diagnosis.
- He et al. (2015), Deep Residual Learning for Image Recognition — residual connections; the structural fix that made real depth possible.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating it as historical. It's managed, not removed, and RNNs over long sequences still hit it.
- Not logging per-layer gradient norms. The diagnosis is one plot away.
- Confusing it with exploding gradients. Exploding crashes loudly; vanishing looks like mediocre training.
- Building a deep network without residuals and blaming the depth.
At a glance
FieldDeep Learning
The mechanismgradients are products of Jacobians; products decay exponentially
Sigmoid's derivativecaps at 0.25, hence the twenty lost years
The structural fixresidual connections, gradient gets a +1 path
The mirror problemexploding gradients, fixed by clipping
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Vanishing vs. exploding gradients — the same arithmetic in two directions. One crashes and is trivially fixed; the other is silent and cost the field twenty years.