How transformers work: the architecture that ate AI

In 2017, a paper with the almost cheeky title "Attention Is All You Need" introduced an architecture called the transformer. Within five years it had displaced nearly every prior approach to processing language, then spread to images, audio, video, protein folding, and code. Every model you've heard of, the GPT series, Claude, Gemini, Llama, is a transformer. It is not an exaggeration to say this single architecture is the substrate of the entire modern AI boom. If you understand how a transformer works, you understand the shape of almost everything happening in AI right now.

So this is the transformer, stripped to the ideas rather than the intimidating diagram from the paper: what problem it solved, why the attention mechanism at its heart was such a leap, why the unglamorous property of parallelism is the real reason it won, how the pieces fit into a working model, and, because it's 2026 and the story has a new chapter, where this dominant architecture is finally facing serious challengers. By the end, the machine under all of modern AI should feel like an idea you own rather than a black box you invoke.

The problem: how do you read a sequence?

To see why the transformer was a breakthrough. You have to feel the problem it replaced, which is the fundamental difficulty of sequence. Language is sequential, the meaning of a sentence depends on the order of its words and on relationships between words that can be far apart. "The dog that chased the cat was tired" requires connecting "dog" to "was tired" across the whole clause about the cat. Any system that processes language has to handle these long-range dependencies.

The pre-transformer answer was the recurrent neural network, and its approach was intuitive: read the sequence one word at a time, left to right, maintaining a running "memory" (a hidden state) updated at each step. It's how you'd read a sentence yourself, so it seems natural. But this approach had two crippling problems that the transformer would solve at a stroke.

First, long-range memory decayed. Information from early in the sequence had to survive being passed through the hidden state at every subsequent step, and it tended to fade, the connection between "dog" and "was tired" weakened with every intervening word. LSTMs and gating mechanisms mitigated this (the vanishing-gradient problem), but never fully solved it. Second, and ultimately more decisive, the sequential processing couldn't be parallelised. Because each step depended on the previous step's output. You had to process word 1, then word 2, then word 3, strictly in order. On modern hardware built to do thousands of things at once, this one-at-a-time constraint was a straitjacket. You couldn't train on enormous datasets in reasonable time, because the architecture refused to be parallelised. Hold onto that, because it's the key to everything.

The core idea: attention

The transformer's central move was to throw out recurrence entirely and replace it with a mechanism called self-attention. Instead of passing information step by step through a hidden state, attention lets every position in the sequence look directly at every other position, all at once, and decide what's relevant.

The intuition. When the model processes the word "was" in "the dog that chased the cat was tired," attention lets it look back at every other word simultaneously and ask, for each: how much does this word matter for understanding "was" right now? It learns to attend strongly to "dog" (the subject that "was tired" refers to) and weakly to "cat" (a distraction inside the relative clause). It pulls in information from the words that matter, weighted by relevance, in a single operation, no passing a fragile memory down a chain. The connection between "dog" and "was" is now direct, a one-step link rather than a signal that had to survive five hops. Long-range dependency, solved.

Mechanically, and this is the one piece of machinery worth knowing, attention works through three roles that each word's vector plays, evocatively named query, key, and value. Think of it like a lookup. Each word issues a query ("what am I looking for?"), and every word offers a key ("here's what I'm about"). The model matches queries against keys to compute relevance scores, how much each word should attend to each other word, and then uses those scores to pull a weighted blend of the values (the actual information each word carries). The word "was" issues a query that matches strongly with the key of "dog," so it pulls in "dog"'s value heavily. That's the entire mechanism: queries find relevant keys, and relevant keys deliver their values. Everything else in a transformer is scaffolding around this one operation.

Why parallelism is the real breakthrough

Attention gets the headlines, but the deeper reason the transformer won is subtler, and most explanations underweight it: attention can be computed in parallel, and recurrence cannot.

Because every word attends to every other word simultaneously rather than waiting for a sequential pass, the whole operation is a set of large matrix multiplications, exactly the computation that modern GPUs are built to do at massive scale, all at once. Where an RNN forced you to process a sequence one step at a time, a transformer processes the entire sequence in parallel. This is what made it possible to train on internet-scale data: the architecture could finally use the full power of the hardware, turning what would have been years of sequential training into something tractable.

And this unlocked the second great discovery of the era: scaling laws. Once you could train enormous models on enormous data, researchers found that transformer performance improved predictably as you added more parameters, more data, and more compute, smoothly, without plateauing, further than anyone expected. The transformer didn't just work better; it kept working better the bigger it got. That scalability, made possible by parallelism, is what turned a clever architecture into the foundation of models with hundreds of billions of parameters. The attention mechanism solved long-range dependencies; parallelism let you scale the solution to the point where startling capabilities emerged. Both mattered, but parallelism is the quieter, more important half of why we're here.

The pieces of a working transformer

Attention is the heart, but a real transformer wraps several components around it. Here's how they fit, in order, and you'll recognise this as the skeleton under any language model.

Input: tokens and embeddings. Text is first split into tokens and each token converted to an embedding, a vector encoding its meaning. This is what attention operates on: not words, but meaning-vectors.

Positional encoding. Here's a subtlety attention creates. Because attention looks at all words simultaneously rather than in order. It has no inherent sense of sequence, to raw attention, "dog bites man" and "man bites dog" look identical. So the transformer adds a positional encoding to each token, a signal marking where in the sequence it sits. Order, which recurrence got for free by processing left to right, has to be explicitly injected back in. Modern variants use simple schemes (rotary position embeddings) to do this well.

Multi-head attention. Rather than computing attention once, a transformer computes it several times in parallel, in different "heads," each learning to attend to a different kind of relationship, one head might track grammatical subject-verb links, another might track which pronoun refers to which noun, another might follow topical similarity. Multiple heads let the model attend to many kinds of relationship at once, then combine them. It's a major part of why transformers are so expressive.

The feed-forward layers and the stack. After each attention step, each position passes through a small neural network that processes the attended information further. And this whole unit, attention plus feed-forward, is stacked, dozens of times. Each layer refines the representation the previous one produced, building from surface patterns in early layers toward abstract meaning and intent in later ones. Depth is where a transformer's sophistication accumulates. (Two engineering pieces make deep stacks trainable: residual connections, which let information skip layers so gradients don't vanish, and layer normalisation, which keeps the signal well-scaled. They're not glamorous, but without them the deep stack wouldn't train.)

Encoder, decoder, or both. The original transformer had two halves, an encoder that reads and understands an input, and a decoder that generates an output, designed for translation. Modern systems usually use one half. Language models like the GPT and Claude families are decoder-only: they're built to generate, predicting the next token over and over. Some understanding-focused models (the BERT lineage) are encoder-only. The core attention machinery is the same; what differs is which half you keep and how attention is masked.

Why this architecture, specifically, changed everything

It's worth pausing on why this particular design had such outsized consequences, because it wasn't obvious in advance that one architecture would dominate so completely.

The transformer turned out to be a general-purpose sequence processor. Attention doesn't care whether the sequence is words, image patches, audio frames, or amino acids, it's a mechanism for letting elements of a set exchange information based on relevance, which is a shockingly universal need. So the same architecture that conquered language went on to power image models (vision transformers treat an image as a sequence of patches), audio, video, and protein structure prediction. One architecture, most of AI. Combined with parallelism-enabled scaling and the emergence of capabilities that appeared as models grew, the transformer became not just an architecture but the architecture, the common substrate that let advances in one domain transfer to others, and let the entire field pour its resources into scaling a single, well-understood design.

The catch, and the 2026 challengers

An honest account has to name the transformer's real weakness, because it's what the current frontier is attacking. Attention's power, every token looking at every other token, is also its curse: the cost grows quadratically with sequence length. Double the context and you roughly quadruple the attention computation. This is why long context is expensive, and why serving a transformer requires a KV cache that grows with every token, consuming large amounts of memory for long documents. The very mechanism that solved long-range dependencies scales badly to very long sequences.

This is where 2026's most interesting architectural story lives. A family of alternatives called state-space models, the best known is Mamba, attacks exactly this weakness. Instead of having every token attend to every other, Mamba maintains a compressed, evolving internal state (a modernised, cleverer descendant of the RNN's hidden state) that it updates as it reads. This makes its cost grow linearly rather than quadratically with sequence length, and at generation time it runs as a pure recurrence with no ever-growing KV cache, dramatically more efficient for very long sequences. In an ironic twist, the field is partly rediscovering the recurrence the transformer threw out, now engineered to be trainable in parallel and to selectively remember what matters.

this is no longer just research. In 2026, hybrid architectures that interleave Mamba-style layers with attention layers have reached production, IBM's Granite models adopted a hybrid Mamba-Transformer design reporting around 70% less GPU memory and roughly double the inference speed on long-sequence tasks, combining Mamba's efficiency with attention's precision. The emerging consensus isn't that the transformer is being replaced wholesale, but that pure attention may not be the final answer, that the future is likely hybrid, keeping attention where its precision matters and using cheaper state-based mechanisms where efficiency dominates. After nearly a decade of total dominance, the transformer finally has serious competition, and the interesting models of the next few years may not be pure transformers at all.

The short version

A transformer processes a sequence by letting every element attend directly to every other element, weighting them by relevance (via query-key-value matching) instead of passing information step by step through a memory. That replaced the sequential recurrent networks that came before, solving long-range dependencies, but the deeper reason it won is that attention computes in parallel, which let transformers scale to internet-sized training and unlock the smooth scaling laws behind modern AI. Wrapped in positional encodings, multi-head attention, and a deep stack of layers, this one architecture generalised from language to nearly every domain. Its weakness, quadratic cost with sequence length, is exactly what 2026's state-space challengers like Mamba target, pushing the field toward hybrid designs.

the transformer won not just because attention reads a sequence well, but because it reads a sequence in parallel, and parallelism is what let a good idea scale into the foundation of modern AI. Understand attention and you understand what the model does; understand parallelism and you understand why it took over. And understand its quadratic cost, and you understand why the next chapter is already being written.

Common questions

What is a transformer in AI? A transformer is a neural network architecture, introduced in 2017, that processes a sequence by letting every element attend directly to every other element based on relevance, rather than reading step by step. Introduced in the paper "Attention Is All You Need," it replaced earlier recurrent approaches and became the foundation of nearly all modern AI, every major language model (GPT, Claude, Gemini, Llama) is a transformer, and the architecture also powers image, audio, and video models.

What is the attention mechanism? Attention is the core mechanism of a transformer: it lets each position in a sequence look at every other position simultaneously and pull in information weighted by relevance. It works through query, key, and value roles, each word issues a query for what it needs, every word offers a key describing what it is, the model matches queries to keys to score relevance, and uses those scores to blend the values. This creates direct connections between distant words, solving the long-range dependency problem that plagued earlier models.

Why did transformers replace RNNs? Two reasons. First, recurrent networks passed information step by step through a memory that decayed over long sequences, weakening long-range connections; attention links distant elements directly. Second and more important, RNNs processed sequences one step at a time and couldn't be parallelised, while attention processes the whole sequence at once as large matrix operations that modern GPUs excel at. That parallelism is what let transformers train on internet-scale data and scale up, which recurrent networks never could.

Why are transformers so good at scaling? Because their core computation (attention) runs in parallel, transformers could be trained on massive datasets using the full power of modern hardware. This unlocked scaling laws: researchers found transformer performance improved smoothly and predictably as they added more parameters, data, and compute, further than anyone expected. The ability to scale, enabled by parallelism, is what turned the architecture into the foundation of models with hundreds of billions of parameters and the capabilities that emerged at that scale.

What is the difference between encoder and decoder transformers? The original transformer had both: an encoder that reads and understands an input, and a decoder that generates an output, designed for translation. Modern systems usually use one half. Decoder-only models (like the GPT and Claude families) are built to generate text by predicting the next token repeatedly. Encoder-only models (like the BERT lineage) focus on understanding tasks. The underlying attention machinery is the same; they differ in which half is kept and how attention is masked.

What is challenging the transformer in 2026? State-space models, most notably Mamba, are the main challengers. They attack the transformer's key weakness: attention's cost grows quadratically with sequence length, making long context expensive. Instead of every token attending to every other, Mamba maintains a compressed, evolving internal state that scales linearly and needs no growing key-value cache, far more efficient for long sequences. In 2026 hybrid Mamba-Transformer models reached production (IBM's Granite reported ~70% less memory use), suggesting the future is likely hybrid rather than a wholesale replacement.

What does GPT stand for, and how does it relate to transformers? GPT stands for Generative Pre-trained Transformer, and the name describes exactly what it is: a transformer architecture (the T), trained first on a large body of text to predict the next token (pre-trained, the P), and used to produce new text (generative, the G). It is a decoder-only transformer, meaning it uses the transformer's attention mechanism to generate text one token at a time. So GPT is not a different architecture from the transformer; it is a specific way of building and training one, and the naming has become shorthand for this family of generative language models.