Transformer
The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.
Reviewed July 10, 2026Stable
When not to use it
- For small, local, structured problems. If only a fixed neighbourhood matters, a convolution is cheaper and often better — most image tasks never needed a transformer.
- On very long inputs with a tight budget. Attention cost grows with the square of length, so a 100k-token input isn't ten times a 10k one, it's roughly a hundred times.
- On tabular data. Gradient-boosted trees still win on spreadsheet-shaped problems, train in seconds, and can explain themselves.
Reach for something else instead
- Convolutional networks where the signal is local and position-invariant — still the efficient choice for many vision and audio tasks.
- State-space models (Mamba and kin) scale linearly with sequence length and are competitive on long sequences, though not yet across the board at frontier quality.
- Sparse or linear attention variants when you genuinely need the length and can't pay the quadratic bill — you trade a little quality for a lot of context.
Read more on the blog
- How transformers work: the architecture that ate AIOne 2017 paper replaced the entire previous approach to sequence modeling and made modern AI possible. Here's what a transformer actually is, why the attention mechanism was such a breakthrough, why parallelism is the real secret, and where, in 2026, the architecture is finally being challenged.
- Where the attention mechanism actually came fromAttention was a fix for a specific engineering failure in 2014, three years before the paper that made it famous. It was not designed as a theory of cognition, and the name was applied afterwards by analogy.
- I spent a month writing down what every AI concept depends on256 concepts, 350 prerequisite links, one rule. I wrote down my predictions first so I could not quietly retrofit the results. Two of them were wrong, and those are the interesting part.
- How AI generates images: from noise to a pictureType a sentence, get an image that never existed. The technology behind it, diffusion, is one of the most simple ideas in modern AI: teach a model to remove noise, then hand it pure static and let it sculpt. Here's how it actually works, how text steers it, and the 2026 rivalry reshaping the field.
The transformer: attention and feed-forward, stacked.
Tokens become vectors, get positional information added (attention alone is order-blind), then pass through a stack of identical blocks. Each block does self-attention (every token looks at every other) followed by a feed-forward step, with residual connections around both. Stack enough blocks and you have the architecture behind every modern LLM.
Further reading
- Vaswani et al. (2017), Attention Is All You Need — the original, and still readable.
- Alammar (2018), The Illustrated Transformer — the explanation most practitioners actually learned from.
- Gu & Dao (2023), Mamba: Linear-Time Sequence Modeling with Selective State Spaces — the most credible challenger to attention's quadratic cost.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming more context is free. Doubling the prompt roughly quadruples attention cost, and can lower answer quality by burying the relevant passage.
- Reading attention weights as an explanation of the model's reasoning. They show where it looked, which is not the same as why it answered — and the field is genuinely divided on this.
- Confusing the transformer with attention. Attention is one mechanism inside the block; the feed-forward layers, residuals, and normalisation are doing real work too.
At a glance
FieldDeep Learning
Introduced2017 — "Attention Is All You Need"
Core ideaSelf-attention
PowersGPT, Claude, Gemini, and most modern AI
Key limitationCost grows with the square of input length
DifficultyIntermediate → Advanced
Often compared with
Transformer vs. CNN — attention over the whole sequence vs. local convolutional filters.
Where this sits
15 concepts come first. Understanding it opens up 70 more.
9Levelsteps in
15Needs firstconcepts
70Opens up23% of 310
8Areascrosses fields
Computed from the prerequisite graph, not assigned. How this works