What is a context window? Why bigger isn't better

Every few months a new model launches with a bigger context window, and the number is presented as a headline feature: a hundred thousand tokens, then two hundred thousand, then a million. The implication is that a bigger window means a smarter, more capable model that can hold your whole codebase or a stack of documents in mind at once. The reality, which the marketing pages leave out, is that models do not use their long context nearly as well as the numbers suggest. A model's advertised context window and the window it can actually reason over reliably are two very different numbers, and the gap between them is one of the most practically important and least understood facts about working with AI.

This piece explains what a context window really is, why bigger has turned out not to mean better, the two distinct ways long context fails (which most explanations confusingly blur together), why the popular test that models ace does not measure what people think it does, and what to do about all of it. Understanding this is what separates people who dump everything into a giant prompt and wonder why the results are mediocre from people who get reliable output, and it reframes a debate that a lot of AI architecture still turns on.

What a context window actually is

The context window is the maximum amount of text a model can consider at once, measured in tokens, and counting both your input and the model's output together. It is the model's working memory for a single request: everything it can "see" while producing an answer, from the system prompt through your question to any documents you have pasted or retrieved. Anything inside the window can, in principle, inform the response; anything outside it does not exist as far as that request is concerned. When people say a model has a 200,000-token window, they mean it can take in roughly 150,000 words of combined input and output before it runs out of room.

That capacity has grown enormously, from a couple of thousand tokens in early models to hundreds of thousands and, at the frontier, a claimed million or more. Taken at face value, that sounds like it should have solved the problem of giving a model enough information. It did not, and the reason is that fitting text into the window is not the same as the model actually using it.

The contradiction the marketing skips

Here is the finding that reframes everything. Frontier models advertise windows of a million tokens and score above 99 percent on the standard long-context test, yet their accuracy on real long-context tasks collapses far below their advertised limits. Careful studies across many frontier models have shown, repeatedly, that performance degrades in non-obvious ways well before the window is full. A model with a 200,000-token window can show serious accuracy loss at a fraction of that. A million-token window does not reliably reason across a million tokens. The advertised size correlates only weakly with how well the model actually uses that space.

To understand why the reassuring benchmark scores and the disappointing real-world results can both be true, you have to look at the test everyone quotes, and then at the two separate ways long context actually breaks.

Why "needle in a haystack" misleads

The popular test is called needle in a haystack: hide one distinctive, out-of-place sentence (the needle) somewhere in a long document (the haystack) and ask the model to find it. Modern models pass this at near-perfect rates, which is where the confident marketing comes from. The problem is that this task is far easier than real long-context work, and passing it says little about the capability people actually need.

Finding a single, semantically distinct sentence is a pure retrieval task with an obvious answer that stands out from its surroundings. Real long-context work is nothing like that. It requires integrating many facts scattered across the context, reasoning over them together, and distinguishing relevant information from a sea of plausible-looking but irrelevant material. When the task shifts from "find the one weird sentence" to "synthesise information from twelve places in this document, some of which look similar to things that do not matter," performance falls apart at far shorter lengths. The needle test measures the easy case and is used to imply competence at the hard case, which is exactly why the benchmark scores and the lived experience diverge so sharply.

Failure mode one: lost in the middle

The two ways long context breaks are distinct, and separating them is the key to understanding the whole subject. The first is lost in the middle, a positional effect first documented by researchers at Stanford. Models attend well to information at the beginning and the end of their context, and poorly to information in the middle. Plot accuracy against where the key fact sits and you get a U-shape: high at the start, high at the end, and a substantial dip, often twenty to thirty points of accuracy, when the important information is buried in the middle.

The practical consequence is stark. If the fact your question depends on sits at, say, position 50,000 of a 200,000-token input, the model may functionally ignore it, not because it ran out of room but because it does not attend well to that region. Where you place information in the context matters as much as whether it is there at all. This alone breaks the intuition that a model reads its context like a person reads a document, evenly, front to back. It does not; it favours the edges.

Failure mode two: context rot

The second failure mode is subtler and, in some ways, more troubling. It is called context rot: the model's accuracy declines simply as the input gets longer, even when the relevant information is fixed, favourably placed, and the window is nowhere near full. This is not about position, like lost-in-the-middle; it is about sheer quantity. More tokens in means worse output out, independent of where the key facts are.

A large study testing eighteen frontier models found that every single one of them degrades as input length grows, at every length increment tested. Not some, not most: all of them. Controlled experiments have shown reasoning accuracy falling steeply, for example from over ninety percent down toward two-thirds, as inputs grew from a few hundred tokens to a few thousand, with the evidence held constant. The window was not full. The information was right there. The model simply got worse at using it because there was more surrounding text. This is the finding the headline "context rot exists" understates: adding context you might think is harmless, or even helpful, actively degrades performance.

Why it happens: attention has a budget

Both failure modes trace back to the same root, which connects to how the transformer architecture works. A model's attention is a finite resource that has to be spread across every token in the context. Add more tokens and each one competes for a share of a fixed attention budget, so every individual piece of information gets less focus. Because attention grows quadratically with length, a longer context means an enormous number of pairwise relationships all diluting each other. That dilution is the engine behind context rot.

Two related effects sharpen the picture. Distractor interference means that semantically similar but irrelevant content actively misleads the model: the more the noise resembles the signal, the harder the model finds it to pick out the answer, and adding such distractors degrades performance beyond what length alone explains. Strikingly, models sometimes do better on a shuffled, incoherent haystack than on a logically coherent document, apparently because coherent text produces more convincing distractors. And at very long context, attention tends to collapse onto a few "sink" tokens, often the very first token and the most recent handful, which leaves the middle effectively invisible and explains why recency and primacy dominate. The common thread is that attention allocation, not storage, is the bottleneck. The model has room for the tokens; what it lacks is the ability to attend to all of them well at once.

Effective context versus advertised context

Put this together and the single most useful concept to carry away is the distinction between advertised context and effective context. The advertised window is the maximum the model will physically accept. The effective window is the length over which it actually maintains reliable retrieval and reasoning, and it is dramatically shorter. Models commonly begin degrading well before their stated limit, often losing significant accuracy at a third to a half of the advertised size, and the drop is frequently sudden rather than gradual, a cliff rather than a slope. The honest way to read a context-window spec is as a ceiling on what fits, not a promise of what works. A million-token window is a container, not a guarantee that the model can think across a million tokens.

What to do about it

The constructive lesson is that with long context, less is often more, and the answer to a hard problem is rarely to pour more text into a bigger window. The practices that work all reduce or organise what goes into the context rather than maximising it. Keep the context clean: include what the task needs and leave out what it does not, because irrelevant tokens do not just waste space, they actively degrade the output through dilution and distraction. Place the most important information at the very start or the very end, where the model attends best, rather than in the middle. And rather than dumping an entire corpus into the window and hoping, use retrieval to pull in only the tightly relevant chunks for each query, which is a large part of why RAG remains valuable even as windows grow, as covered in the pieces on how RAG works and RAG versus fine-tuning. This deliberate curation of the context is exactly the discipline of context engineering, and it matters precisely because more context is not free. The industry itself is shifting its attention from ever-larger windows toward better management of the windows we have, because the returns on raw size have turned out to be so limited.

The short version

A context window is the maximum text a model can consider at once, measured in tokens and counting input plus output. Windows have grown to a million tokens or more, but models do not use long context well, so the advertised window and the window a model can reliably reason over are very different sizes. The popular needle-in-a-haystack test, which models ace, measures easy single-fact retrieval and overstates real capability, which requires integrating many facts. Long context fails in two distinct ways: lost-in-the-middle, where models attend to the start and end but not the middle, and context rot, where accuracy declines simply as input grows even when the window is not full. Both come from attention being a finite budget spread across all tokens. The practical response is to keep context clean, place key information at the edges, and retrieve only what is relevant rather than dumping everything in.

The idea to hold onto is that a context window is a container, not a memory, and filling it does not mean the model can use what is inside, because a longer context spreads the model's finite attention thinner and buries the middle, so effective context is far shorter than the advertised number. Bigger windows are a real convenience, but the belief that you can solve a problem by pouring more text into a larger one is the single most common and costly misconception about working with these models.

Common questions

What is a context window in an LLM? A context window is the maximum amount of text a model can consider at once, measured in tokens and counting both the input (system prompt, your question, any pasted or retrieved documents) and the model's output together. It is the model's working memory for a single request: everything it can see while producing an answer. Anything inside the window can inform the response; anything outside it effectively does not exist for that request. A 200,000-token window holds roughly 150,000 words of combined input and output.

Does a bigger context window make a model better? Not as much as the numbers imply. Fitting more text into the window is not the same as the model using it well, and research consistently shows performance degrading well before the window is full. A model's advertised window (what it will accept) and its effective window (what it can reliably reason over) are very different, with the effective one dramatically shorter. Bigger windows are a real convenience for fitting more in, but they do not deliver proportional gains in the model's ability to actually reason across all that text.

What is the "lost in the middle" problem? Lost in the middle is a positional effect where models attend well to information at the beginning and end of their context but poorly to information in the middle. Plotting accuracy against where the key fact sits produces a U-shape, high at the edges and dropping twenty to thirty points when the important information is buried in the middle. The practical consequence is that a fact sitting in the middle of a long input can be functionally ignored, so where you place information in the context matters as much as whether it is present.

What is context rot? Context rot is the decline in a model's output quality simply as its input grows longer, even when the relevant information is fixed, well-placed, and the window is nowhere near full. Unlike lost-in-the-middle, which is about position, context rot is about sheer quantity: more tokens in produces worse output out. A study of eighteen frontier models found every one degrades as input length increases, with reasoning accuracy dropping substantially as inputs grew even a few thousand tokens. It means adding context you think is harmless can actively hurt performance.

Why do long context windows degrade at all? Because a model's attention is a finite resource spread across every token in the context. Add more tokens and each competes for a share of a fixed attention budget, so every piece of information gets less focus, an effect amplified because attention grows quadratically with length. Semantically similar but irrelevant content (distractors) further misleads the model, and at very long lengths attention tends to collapse onto a few tokens at the start and end, leaving the middle effectively invisible. The bottleneck is attention allocation, not storage: the model has room for the tokens but cannot attend to all of them well.

Why does "needle in a haystack" overstate long-context ability? Because it tests the easy case. Needle-in-a-haystack hides one distinctive, out-of-place sentence in a long document and asks the model to find it, which is pure retrieval of an answer that stands out. Real long-context work requires integrating many facts scattered through the context, reasoning over them together, and separating relevant information from plausible-looking noise. Models pass the needle test at near-perfect rates while failing the harder integration tasks at much shorter lengths, which is why the benchmark scores look great but real-world long-context results disappoint.

How should I handle long context in practice? Treat less as more. Keep the context clean by including only what the task needs, since irrelevant tokens degrade output through dilution and distraction rather than helping. Put the most important information at the very start or end, where models attend best, not in the middle. And instead of dumping an entire corpus into a large window, use retrieval (RAG) to pull in only the tightly relevant chunks for each query. This deliberate curation, known as context engineering, produces far more reliable results than relying on a big window to sort through everything for you.