Activation Function
The small non-linear function after each layer — without it a hundred-layer network collapses into a single line.
When not to use it
- (You need one. The question is which.)*
- Sigmoid/tanh in deep hidden layers. Gradients vanish. This is a solved historical mistake, not a preference.
- Softmax as a hidden activation. It's a distribution over a vector, for outputs.
- An exotic activation from a recent paper. Gains are small, inconsistent, and rarely replicate on your architecture.
Reach for something else instead
- ReLU — CNNs, and anything where you're not sure.
- GELU — transformers. What your model uses.
- SwiGLU — modern LLMs. More parameters, better results, no explanation.
- Leaky ReLU — if dying units are actually your problem, which they usually aren't.
Sources & further reading
- Nair & Hinton (2010), Rectified Linear Units Improve Restricted Boltzmann Machines — where ReLU enters.
- Glorot, Bordes & Bengio (2011), Deep Sparse Rectifier Neural Networks — why the simplest option beat the elegant ones.
- Shazeer (2020), GLU Variants Improve Transformer — SwiGLU, and an unusually honest admission that the explanation is absent.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Thinking depth alone buys expressiveness. Without a non-linearity, a hundred layers is one layer.
- Using sigmoid in hidden layers because it's the one from the textbook diagram.
- Tuning the activation function. It's a solved default; your time is better spent on the learning rate.
- Expecting a principled reason for GELU or SwiGLU. There isn't one — they were found, then explained.
At a glance
FieldDeep Learning
Why it existsstacked linear layers collapse to one
DefaultReLU for CNNs, GELU/SwiGLU for transformers
Historical failuresigmoid/tanh vanishing gradients
Known bugdying ReLU
How choices were madeempirically, then rationalised
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
ReLU vs. sigmoid — one is
max(0,x) and made deep learning possible; the other is smooth, elegant, biologically motivated, and held the field back for twenty years.