Positional Encoding

The full account

The problem attention created and then had to solve

Attention has a strange property that's easy to miss: it is order-blind. A transformer's attention mechanism, left to itself, treats a sentence as a bag of words — it computes how much each token should attend to each other token, but nothing in that computation knows which token came first. "The dog bit the man" and "The man bit the dog" contain identical tokens, and to raw attention they are the same set. This is not a small bug. For language, where order carries much of the meaning, it's fatal — and positional encoding is the machinery bolted on to fix it.

The fix is to inject information about where each token sits, so that the representation the model works with encodes not just "which word" but "which word, in which position". Get this right and the model can distinguish the dog that bit from the dog that was bitten. Get it wrong — or leave it out — and the most powerful sequence model ever built cannot tell a sentence from its own anagram.

From fixed waves to learned rotations

The original transformer used a fixed, clever scheme: sinusoidal functions of different frequencies, so each position gets a unique signature and the model can, in principle, learn to read relative distances from the interference patterns between frequencies. It worked, it needed no training, and it had a hoped-for bonus — that it might extrapolate to sequences longer than any seen in training, since the functions are defined at any position.

That hope was mostly disappointed, which drove years of iteration. Learned positional embeddings (just train a vector per position) were simpler and often better in-distribution but couldn't extrapolate past their trained length at all. The current frontier, rotary position embeddings (RoPE), took a different route: instead of adding a position signal to the token, rotate the query and key vectors by an angle that depends on position, so that the attention score between two tokens naturally depends on their relative offset. RoPE became close to standard in modern LLMs because it handles relative position elegantly and degrades more gracefully as sequences grow.

Why this is really about long context

The reason positional encoding is an active research area in 2026 rather than a solved detail from 2017 is that it turns out to be the bottleneck for long context. A model trained on 4,000-token sequences doesn't automatically work at 100,000 tokens, and why it fails is largely a positional-encoding story: the position signals at the far end are unlike anything seen in training, and the model's behaviour there is undefined. The techniques that extend context windows — position interpolation, frequency scaling methods like NTK-aware and YaRN — are almost all interventions on the positional encoding, stretching or reshaping the position signal so the model can operate at lengths it never trained on.

So the humble order-fixing bolt-on turns out to sit on the critical path of one of the field's most commercially important frontiers. Every "million-token context" announcement is, under the hood, substantially a claim about having solved a positional-encoding extrapolation problem.

The lesson in the bolt-on

Positional encoding is a good teacher of a general truth about deep learning: the components that look like incidental plumbing often encode the deepest assumptions. Attention's order-blindness is a feature — it's what lets attention be parallel and permutation-flexible — and positional encoding is how you buy back the order you deliberately threw away. The design of that buy-back determines how well the model handles length, structure, and distances it never saw. A detail introduced in a single paragraph of the original paper became, a decade later, one of the load-bearing walls of the whole architecture.