Context Window
The maximum amount of text an AI can consider at once — its short-term working memory, measured in tokens.
When not to use it
- As a substitute for retrieval. Pasting a whole knowledge base into a huge window is expensive on every call and often less accurate than fetching the right three paragraphs.
- Filling it because it's there. Models attend unevenly across long contexts — material buried in the middle gets used less reliably than material at either end.
Reach for something else instead
- RAG when the source material is bigger than the window, or changes, or only a fraction is relevant.
- Summarise-then-reason — compress earlier turns rather than resending everything.
- Caching repeated context, so the same preamble isn't billed on every request.
Read more on the blog
- Context engineering: the skill that replaced prompt engineeringPrompt engineering didn't die, it got absorbed. The bottleneck moved from how you phrase a request to what information surrounds it. Here's what context engineering actually is, why it took over in 2026, and the discipline underneath the buzzword.
- What is a context window? Why bigger isn't betterEvery model launch brags about a bigger context window, now measured in millions of tokens. The part the marketing leaves out is that models do not use long context well. A model's advertised window and the window it can actually reason over are very different numbers, and the gap explains a lot of real-world AI failures.
- How AI memory works, and why the model remembers nothingAsk whether a chatbot remembers you and the honest answer surprises people: the model itself remembers nothing. Language models are stateless, starting every conversation from zero. Everything users experience as memory is an engineering layer bolted around the model that re-feeds or retrieves text. Here is how that layer works, why the model is built to forget, and where memory breaks.
- Why long context windows fail: the lost-in-the-middle problemModels now advertise context windows of a million tokens, and they pass the standard needle-in-a-haystack test almost perfectly. Yet their accuracy on real long-context work collapses far below those advertised limits. The gap between what a model will accept and what it can actually use is one of the most practically important facts about modern AI.
The answer shares the window.
A context window is a single budget covering everything the model reads and everything it writes — the prompt does not sit beside the reply, it competes with it. Fill 95% of a window with retrieved documents and there is no room left for the answer you retrieved them for, which surfaces as a reply that stops mid-sentence rather than as an error. The failure is silent, which is why it survives so long in production. Everything here is subtraction against the window size you pick.
The full account
The number on the box is a capacity, not a promise
A model advertised at 128K tokens will accept 128K tokens without throwing an error. That is the entire claim the number makes. It is not a claim that the model attends to those tokens evenly, or that it can find what's in the middle of them, or that performance at 128K resembles performance at 8K. Those are separate questions, and the answers are worse than the marketing implies.
Two results settle this, and they are the two you should know before paying for a longer window.
Lost in the middle
Liu and colleagues in 2023 asked a simple question: does where you put the relevant information in a long context change whether the model uses it? It should not. Attention is nominally symmetric over positions, and nothing in the architecture says position 5,000 is worse than position 50.
It does change, and substantially. Performance is highest when the relevant information sits at the very beginning or the very end of the input, and degrades markedly when it sits in the middle — a U-shaped curve, present even in models explicitly built and marketed for long context. The effect is large enough that a model can fail to use a document it was given, purely because of where in the stack it landed.
This turns position into a design variable. If your retrieval system returns ten chunks and puts the best match fifth, you have put your best evidence in the model's blind spot. Ranking now determines not just what is included but where — and where determines whether it is read.
RULER, and the gap between claimed and real
Hsieh and colleagues built the benchmark the field needed in 2024, and its title is the question: what's the real context size?
The needle-in-a-haystack test, which everyone was using, hides one distinctive sentence in a long document and asks the model to find it. RULER keeps that and adds what it was missing: multi-hop tracing — follow a chain of references across the context — and aggregation — combine information spread through it. Then it evaluates 17 long-context models across 13 tasks at increasing lengths.
The result is the one to quote. All 17 models claim context sizes of 32K or greater. Only about half maintain satisfactory performance at 32K. And they do this while scoring near-perfect on the vanilla needle test at the same lengths. The retrieval score stays green; everything else falls over.
So "effective context length" is a real and much smaller number than the advertised one, and the gap widens with task complexity rather than with length alone. A model can genuinely hold 200K tokens, genuinely find a planted sentence anywhere in them, and genuinely fail to answer a question that requires connecting two facts 100K apart.
| What it tests | What models score | |
|---|---|---|
| Needle-in-a-haystack | find one distinctive planted sentence | near-perfect — it's close to string matching |
| RULER: multi-hop tracing | follow a chain of references | large drops as length rises |
| RULER: aggregation | combine information spread across the window | large drops as length rises |
| Claimed vs effective | — | all 17 claim ≥32K; ~half hold at 32K |
The arithmetic your figure computes
There is a second constraint that is pure bookkeeping and catches people constantly: the window holds the question and the answer.
The figure computes it. Fill the context with your prompt and retrieved documents and you have not merely used most of the budget — you have determined how much room is left for the reply. Push the input from a comfortable fraction to nearly full and the answer space collapses from about 7,800 tokens to 82. The model does not warn you. It truncates, or it produces something clipped, and the failure looks like a quality problem rather than an accounting one.
This compounds with everything above. Longer input means less answer room, worse middle-of-context recall, quadratic attention cost, and a larger KV cache — all moving the wrong way together, in exchange for information the model may not read.
The strategic question
The honest framing is that long context and retrieval are competing answers to the same problem, and the industry has an incentive to prefer the one it bills for. Give the model everything and let attention sort it out; or find the right thing first and give it only that.
RULER and lost-in-the-middle both argue for retrieval, and not on cost grounds. A well-ranked handful of chunks placed where the model actually attends will frequently beat a 128K stuff, because you have done the selection the model does badly. The counter-argument is real too: retrieval fails when you don't know what to retrieve, and there are tasks — summarising a whole document, tracing a theme across a book — where selection is the thing you're trying to do.
The position that survives both papers: the window is a capacity you should try not to fill. Measure your effective context on your own tasks, not the vendor's number. Put the important material at the start or the end. And treat every token of input as a token of answer you gave away.
Further reading
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts — models use the beginning and end more reliably than the middle.
- Press, Smith & Lewis (2022), Train Short, Test Long (ALiBi) — one of the position-encoding tricks that made longer contexts feasible.
- Dao et al. (2022), FlashAttention — why long contexts got cheaper without changing the maths.
- Hsieh et al. (2024), RULER: What's the Real Context Size of Your Long-Context Language Models? — 17 models all claiming 32K+; only about half hold up at 32K once the task is more than retrieval.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Forgetting the answer shares the window. A prompt that nearly fills the context leaves no room to respond, and output gets truncated.
- Assuming a bigger window means better recall over that window. Longer context reliably costs more; it does not reliably work better.
- Measuring context in words or characters. It's tokens, and the conversion is not what you think for code, numbers, or non-English text.
At a glance
Where this sits
24 concepts come first. Understanding it opens up 2 more.
Computed from the prerequisite graph, not assigned. How this works