RNN (Recurrent Neural Network)
A network that reads a sequence one step at a time, carrying a memory forward — the obvious way to handle language, and the reason it took so long to work.
When not to use it
- For anything you'd use a transformer for. It's sequential, so you can't parallelise training, and that's the whole ballgame on modern hardware.
- On long dependencies, in vanilla form. The memory decays geometrically. That's the point of LSTMs.
- When you can see the whole sequence. Attention connects every position directly. Recurrence makes you walk there.
Reach for something else instead
- Transformer — parallel, direct connections, quadratic cost. What won.
- LSTM / GRU — recurrence with additive gates so the gradient survives.
- State-space models (Mamba) — recurrence done properly. Linear cost, constant state, competitive.
- 1D convolutions — for local patterns in sequences, often enough and fully parallel.
Sources & further reading
- Elman (1990), Finding Structure in Time — the simple recurrent network; where the idea gets its modern form.
- Bengio, Simard & Frasconi (1994), Learning Long-Term Dependencies with Gradient Descent is Difficult — the proof that it's structural, not a training bug.
- Gu & Dao (2023), Mamba: Linear-Time Sequence Modeling with Selective State Spaces — recurrence, rebuilt properly.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Thinking transformers won on modelling elegance. They won on parallelism, which is a hardware fact.
- Using a vanilla RNN for long sequences. It cannot retain the information; that isn't a tuning issue.
- Forgetting truncated BPTT caps what the model can learn. Dependencies longer than the window are invisible.
- Treating recurrence as dead. State-space models are recurrence, and they're a live contender.
At a glance
FieldDeep Learning
Idearead one step at a time, carry a hidden state
Two fatal flawssequential (no parallelism), memory decays geometrically
The mathsgradient involves Uᵏ; below 1 vanishes, above 1 explodes
Statusreplaced, and returning as state-space models
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
RNN vs. transformer — one walks through the sequence carrying a memory; the other connects every position directly. The second is parallel, which decided it.