RNN (Recurrent Neural Network)
A network that reads a sequence one step at a time, carrying a memory forward — the obvious way to handle language, and the reason it took so long to work.
Reviewed July 12, 2026Stable
When not to use it
- For anything you'd use a transformer for. It's sequential, so you can't parallelise training, and that's the whole ballgame on modern hardware.
- On long dependencies, in vanilla form. The memory decays geometrically. That's the point of LSTMs.
- When you can see the whole sequence. Attention connects every position directly. Recurrence makes you walk there.
Reach for something else instead
- Transformer — parallel, direct connections, quadratic cost. What won.
- LSTM / GRU — recurrence with additive gates so the gradient survives.
- State-space models (Mamba) — recurrence done properly. Linear cost, constant state, competitive.
- 1D convolutions — for local patterns in sequences, often enough and fully parallel.
Read more on the blog
- Where the attention mechanism actually came fromAttention was a fix for a specific engineering failure in 2014, three years before the paper that made it famous. It was not designed as a theory of cognition, and the name was applied afterwards by analogy.
- How transformers work: the architecture that ate AIOne 2017 paper replaced the entire previous approach to sequence modeling and made modern AI possible. Here's what a transformer actually is, why the attention mechanism was such a breakthrough, why parallelism is the real secret, and where, in 2026, the architecture is finally being challenged.
- What is deep learning? The complete guideAlmost every AI system that impresses today, from chatbots to image generators to voice assistants, runs on deep learning. Its core idea is a single powerful shift: instead of humans hand-crafting the features a model uses, deep networks learn their own layered representations of data, from simple edges to whole objects. This guide explains what deep learning is, how it works, its architectures, and its limits.
- What is natural language processing (NLP)?Every search, translation, voice assistant, and chatbot runs on natural language processing, the field of getting computers to work with human language. Its defining modern story is a quiet revolution: the dozens of separate, task-specific methods that made up NLP for decades collapsed into a single general approach, the large language model.
An RNN reads a sequence one step at a time, passing a memory forward.
Unlike a feedforward network, a recurrent network has a loop: at each step it takes the current input and its own previous hidden state, so information from earlier in the sequence carries forward. This is what lets it handle text, audio, and time series — but the same loop is why long sequences cause vanishing gradients, the problem LSTMs were built to fix.
Further reading
- Elman (1990), Finding Structure in Time — the simple recurrent network; where the idea gets its modern form.
- Bengio, Simard & Frasconi (1994), Learning Long-Term Dependencies with Gradient Descent is Difficult — the proof that it's structural, not a training bug.
- Gu & Dao (2023), Mamba: Linear-Time Sequence Modeling with Selective State Spaces — recurrence, rebuilt properly.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Thinking transformers won on modelling elegance. They won on parallelism, which is a hardware fact.
- Using a vanilla RNN for long sequences. It cannot retain the information; that isn't a tuning issue.
- Forgetting truncated BPTT caps what the model can learn. Dependencies longer than the window are invisible.
- Treating recurrence as dead. State-space models are recurrence, and they're a live contender.
At a glance
FieldDeep Learning
Idearead one step at a time, carry a hidden state
Two fatal flawssequential (no parallelism), memory decays geometrically
The mathsgradient involves Uᵏ; below 1 vanishes, above 1 explodes
Statusreplaced, and returning as state-space models
DifficultyIntermediate
Often compared with
RNN vs. transformer — one walks through the sequence carrying a memory; the other connects every position directly. The second is parallel, which decided it.
Where this sits
4 concepts come first. Understanding it opens up 80 more.
4Levelsteps in
4Needs firstconcepts
80Opens up26% of 310
8Areascrosses fields
Learn these firstNeural Network
Computed from the prerequisite graph, not assigned. How this works