Positional Encoding
How a transformer knows what order the words came in — a patch for the architecture's blindness to sequence, and the thing that decides how far context can stretch.
When not to use it
- (You need something. The question is which, and how far to trust it.)*
- Learned absolute encodings, if you'll ever exceed the training length. They cannot extrapolate — there's no embedding for a position you never trained.
- Naive RoPE far past training length. Without interpolation or scaling, quality degrades in ways that don't announce themselves.
- ALiBi, if long-range attention is the point. The recency prior is a feature for most language and a bug for retrieval over long documents.
Reach for something else instead
- RoPE — the current default, and what almost everything uses.
- ALiBi — better native extrapolation, at the cost of a recency bias.
- YaRN / NTK-aware scaling — how existing models get longer context without retraining.
- No positional encoding — apparently viable in decoder-only models, because the causal mask leaks position.
Sources & further reading
- Vaswani et al. (2017), Attention Is All You Need — sinusoidal encodings; the original patch for order-blindness.
- Su et al. (2021), RoFormer: Enhanced Transformer with Rotary Position Embedding — RoPE, and why relative position falls out of a rotation.
- Press, Smith & Lewis (2021), Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — ALiBi; extrapolation by having nothing to extrapolate.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming a long context window means good long-context performance. Usually it's interpolation plus a brief fine-tune, and attention may not reach the far end.
- Using learned absolute encodings then needing extrapolation. That door was closed at training time.
- Treating positional encoding as a solved implementation detail. It's the component that caps your context.
- Reading "128k context" as a capability claim rather than a spec. Test where attention actually degrades.
At a glance
FieldDeep Learning
Problem it solvesattention is order-blind
Current standardRoPE
Property that mattersextrapolation past training length
How long context is really madeinterpolation plus fine-tune
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
RoPE vs. ALiBi — one encodes relative position by rotation, the other by penalising distance. RoPE is more expressive; ALiBi extrapolates more naturally and assumes recency matters.