Why long context windows fail: the lost-in-the-middle problem

Frontier models now advertise context windows measured in the millions of tokens, enough to hold an entire codebase, a legal archive, or a small library in a single prompt. They also pass the standard test for long-context ability, the needle-in-a-haystack search, at better than ninety-nine percent. Both facts are true, and together they suggest that the old problem of a model forgetting what you told it has been solved by sheer capacity. It has not. On real long-context work, accuracy falls apart well below the advertised limit, and it does so in ways that a headline window size gives no hint of. A context window is a capacity, not a competence: the advertised number is the maximum input the model will accept, not the amount it can reliably reason over, and performance degrades both according to where information sits in the input and according to how much surrounds it, which is why filling a large window is usually worse than curating a small one.

This guide explains the two distinct failure modes that get routinely conflated, why the standard benchmark flatters models so badly, the mechanisms in the architecture that cause the decay, the counterintuitive finding that well-structured documents hurt more than jumbled ones, and why retrieval did not become obsolete when windows got huge. The practical upshot is a change in instinct: with long context, the question stops being how much you can fit and becomes how little you can get away with.

The promise, and the reality

The rise of long context was fast. In late 2023 the first models with roughly a hundred thousand tokens of context arrived, and windows have grown by orders of magnitude since, into the millions. The pitch was straightforward and appealing: stop engineering around the model's memory limits and simply give it everything. Put the whole document in, the whole codebase, the whole conversation history.

The evidence that this does not work as advertised is now substantial and consistent. Controlled studies across the leading models of the current generation, including the frontier systems from every major lab, find that output quality degrades measurably as input grows, and that it does so at every length increment tested rather than only near the stated limit. A model with a million-token window does not sail smoothly to nine hundred thousand and then fall off a cliff; it starts losing accuracy far earlier, sometimes at a small fraction of its capacity. The degradation is often abrupt rather than a gentle slope, which makes it harder to anticipate. This phenomenon has acquired a name, context rot, and it applies to every model tested, which suggests it is a property of how these systems are built rather than a shortcoming that the next training run will fix.

Two different failures, routinely confused

Most discussion of long context blurs together two failure modes that behave differently and call for different responses. Separating them is the single most useful move in understanding the problem.

The first is positional degradation, better known as the lost-in-the-middle effect. Here, accuracy depends on where the relevant information sits in the input. Performance follows a U-shape: models attend well to material at the beginning and at the end of the context and much less well to material in the middle. The original study of this, on multi-document question answering, found accuracy dropping by more than thirty points when the needed information was placed in the middle of the context rather than at the start. The model has the information; it simply underweights it because of position. Notably, this can mean a model performs worse with a document in its context than with no document at all, if the relevant passage lands in the dead zone.

The second is length degradation, which is context rot proper. Here, accuracy declines as the input grows even when the relevant evidence is held fixed and placed in a favorable position. This is the more unsettling of the two, because it cannot be fixed by rearranging the input. One controlled study found reasoning accuracy falling from around ninety percent to below seventy as inputs grew from a few hundred tokens to a few thousand, a scale far below any advertised limit. Another found most tested models dropping below half their short-context score by around thirty thousand tokens. Adding tokens costs accuracy even when those tokens are not where the answer lives.

The practical difference matters. Positional degradation says: put the important material at the edges. Length degradation says: do not include material you do not need. The first is a layout problem, the second a volume problem, and a system that only addresses one will still fail on the other.

Why the needle test flatters

If models degrade this reliably, why do they score so well on the standard long-context benchmark? Because that benchmark asks for something much easier than real work. The needle-in-a-haystack test hides a single distinctive sentence inside a long body of unrelated text and asks the model to find it. The needle is deliberately conspicuous, semantically unlike everything around it, and the task is retrieval of one quotable fact.

Real long-context tasks look nothing like this. They ask a model to reason across many scattered passages that are all about the same subject, distinguishing the relevant one from a crowd of plausible near-misses, and to combine information rather than locate it. That difference turns out to be decisive: when the target is semantically distinct from its surroundings, models find it easily, but as the surrounding material becomes harder to tell apart from the answer, accuracy drops sharply, and the drop worsens with length. Semantic similarity, in other words, drives the decay more than raw length does.

This is a textbook case of the evaluation problem in miniature. The needle test measures a narrow proxy, one-fact retrieval from dissimilar text, and gets read as evidence of a much broader ability, competent reasoning over long inputs. Passing it at ninety-nine percent tells you the model can find a sentence. It tells you very little about how the model will behave at two hundred thousand tokens of real work, and treating the two as equivalent is how a benchmark result becomes a marketing claim.

Three mechanisms behind the decay

Three compounding factors explain what is going wrong, and all trace back to the architecture.

The first is attention dilution. The self-attention mechanism at the heart of the transformer, which lets a model relate every token to every other token, works by distributing a finite amount of attention across the input. As the input grows, that budget is spread thinner. A passage that received strong attention in a one-thousand-token context may be functionally ignored inside a hundred-thousand-token one, not because the model cannot see it but because its share of attention has become vanishingly small. The quadratic structure of attention compounds this, since the number of pairwise relationships the model must weigh grows with the square of the input length, so a hundred thousand tokens implies billions of pairwise comparisons among which the signal must compete.

The second is positional bias. Models attend disproportionately to the beginning and end of their input, a tendency arising from both the way position is represented in the architecture and the structure of the text they were trained on, where openings and conclusions carry disproportionate importance. This produces the U-shaped curve directly.

The third is distractor interference. Content that is semantically similar to the answer but irrelevant does not merely take up space; it actively misleads. Studies find that adding a single strong distractor measurably lowers accuracy and that adding several compounds the effect. Because embeddings place similar meanings close together, near-miss passages compete directly with the correct one for the model's attention, and the model has no reliable way to arbitrate between them at scale.

The finding that should change how you build

Among these results is one that runs against nearly everyone's intuition, and it is worth stating plainly. When researchers compared model performance on a long input that was logically coherent and well-organized against the same content shuffled into an incoherent jumble, the models did better on the shuffled version. This held across all eighteen frontier models tested.

The natural assumption is the opposite: that a clean, well-structured hundred-thousand-token document should be easier for a model to reason over than a disordered pile of the same material. The likely explanation is that coherence is exactly what produces good distractors. In a well-written document, every section is topically related to every other, so a question about one part finds many plausible-looking competitors elsewhere in the text. In a shuffled jumble, the correct passage stands out from its neighbors and is easier to isolate. Structure, which helps human readers enormously, costs machine accuracy at scale.

The lesson is not to shuffle your documents. It is that a model's difficulty is driven by how hard the answer is to distinguish from everything else in the context, not by how well the context is organized for a human. That reframes the whole task of preparing input: you are not writing for a reader who follows an argument, you are reducing competition for a mechanism that weighs everything against everything.

Advertised length versus usable length

The gap between these two numbers is the practical takeaway. A model's advertised context window is a hard ceiling on what it will accept without error. Its usable context, the range within which you can trust its output, is substantially smaller, and evidence suggests models commonly become unreliable well before their stated limits. There is no published number that applies universally, because the usable range depends on the task, the difficulty of distinguishing the answer from surrounding material, and how much reasoning rather than retrieval the task requires.

That variability is itself the point. Because the usable range is task-dependent, you cannot read it off a spec sheet, which means it has to be measured for your own workload rather than assumed from the advertised figure. The discipline that follows is straightforward: test at the lengths you actually use, with material as similar as your real data, and find where your own accuracy starts to fall rather than trusting the ceiling printed on the box.

Why bigger windows did not kill retrieval

When million-token windows arrived, a common prediction was that retrieval-augmented generation would become unnecessary. If the whole corpus fits in the prompt, why bother retrieving? Context rot answers that question. Filling a window with everything you have does not give the model access to all of it in any meaningful sense; it dilutes attention, introduces distractors, and buries some of the material in the positions the model weights least. More context is not more knowledge, and past a point it is less.

So retrieval did not die; its job changed. It used to exist because models could not physically hold much text. Now it exists because models cannot effectively use much text, which is a different constraint with a similar remedy. The prevailing pattern is a hybrid: retrieve the material that is actually relevant, which may still be a large amount, and then let the model reason over that curated subset within its long window. The window's real value is not that it lets you skip the curation step but that it raises the ceiling on how much curated material you can reason over at once.

What this means in practice

Several practical instincts follow, and they mostly point the same direction. Treat context as a scarce resource even when the window is large, because every irrelevant token both dilutes attention and risks acting as a distractor. Prefer curation to inclusion, since removing near-miss material often helps more than adding supporting material. Place what matters most at the beginning or the end of the input rather than the middle. Be especially careful with content that is topically similar to the answer, since that is where interference is strongest. And measure degradation on your own task rather than trusting either the advertised window or a headline benchmark score. The general skill here, sometimes called context engineering, amounts to deciding what deserves to be in front of the model at all, which is closer to editing than to prompting.

The honest state of things

Context rot appears to be an architectural property of attention-based models rather than a gap that more training will close, since it shows up in every frontier model tested and at every length. That does not make it permanent. Work on positional encoding, attention variants, retrieval-augmented and memory-augmented designs, and compression all aim at the problem, and models have improved at long context even as the underlying pattern persists. What has clearly shifted is the framing. The industry spent several years competing on window size, a number that is easy to advertise and easy to misread, and the more useful question turned out to be how much of a window a model can actually use. Expect the headline numbers to matter less over time and effective context, measured on real tasks, to matter more. Until then, the safest assumption is that a model's advertised window tells you what it will accept, and only your own testing tells you what it will understand.

The short version

A context window is the maximum amount of input a model will accept, not the amount it can reliably reason over, and modern models degrade well before their advertised limits. Two distinct failures are usually conflated. Positional degradation, the lost-in-the-middle effect, means accuracy depends on where information sits, following a U-shape in which the middle of a long input is heavily underweighted, with drops of thirty points or more compared to the edges. Length degradation, or context rot, means accuracy declines as input grows even when the evidence is held fixed and favorably placed, and it appears at every length increment across every frontier model tested. The standard needle-in-a-haystack benchmark flatters models because it asks them to find one semantically distinctive sentence, whereas real tasks require reasoning across many similar passages, and semantic similarity drives decay more than length does. Three mechanisms cause this: attention dilution as a finite attention budget spreads across more tokens, positional bias toward the start and end, and distractor interference from similar but irrelevant content. Counterintuitively, coherent well-structured documents degrade performance more than shuffled ones, because coherence creates better distractors. This is why retrieval did not become obsolete: the job changed from fitting text into a small window to curating what deserves to be in a large one.

The idea to hold onto is that a context window is a capacity, not a competence, so the advertised number tells you what a model will accept rather than what it can use, and because accuracy falls both with position and with volume, the winning move is almost always to put less in front of the model rather than more. With long context, curation beats inclusion, and the only reliable measure of usable context is the one you take on your own task.

Common questions

Why do large language models fail with long context? Because a context window is a limit on what a model will accept, not a guarantee of what it can use. Three architectural factors cause degradation. Attention dilution: the model's finite attention is spread across more tokens as input grows, so material that was strongly attended in a short context becomes functionally ignored in a long one. Positional bias: models attend more to the beginning and end of an input than the middle. Distractor interference: content that is semantically similar to the answer but irrelevant actively competes with and misleads the model. Together these mean accuracy declines well before the advertised limit, and the effect appears in every frontier model tested.

What is the lost-in-the-middle problem? Lost in the middle is the finding that a model's accuracy depends on where relevant information sits within its context. Performance follows a U-shaped curve: models use information at the start and end of the input well and information in the middle much less well. In the original multi-document question-answering study, accuracy dropped by more than thirty points when the needed passage was placed in the middle rather than at the beginning. The information is present and within the model's window, but it is underweighted because of position. This is why placing important material at the edges of a long prompt measurably improves results.

What is context rot? Context rot is the measurable decline in output quality as input length increases, even when the relevant evidence is held constant and placed in a favorable position. It is distinct from the lost-in-the-middle effect, which is about position rather than volume. Controlled testing across eighteen frontier models found degradation at every input-length increment tested rather than only near the stated limit, meaning a model with a very large window can already be losing accuracy at a small fraction of that window. Because it appears across all tested models, context rot looks like a property of attention-based architectures rather than a flaw specific to any one system.

If a model has a million-token context window, can I just use all of it? You can submit that much, but you should not expect reliable reasoning over it. The advertised window is a hard ceiling on accepted input, while the usable range, where output stays trustworthy, is substantially smaller and depends on the task. Filling a large window dilutes attention across more tokens, buries some material in the positions the model weights least, and introduces irrelevant content that can actively mislead. Models commonly become unreliable well before their stated limits. The practical approach is to measure where accuracy degrades on your own workload rather than assuming the advertised number is usable capacity.

Does the needle-in-a-haystack test prove a model handles long context well? No. That test hides one distinctive sentence in a long body of unrelated text and asks the model to find it, which is retrieval of a single conspicuous fact from material it is easy to distinguish from. Real long-context tasks require reasoning across many scattered passages that are all topically similar, where telling the relevant one apart from plausible near-misses is the actual difficulty. Models can pass the needle test at better than ninety-nine percent while failing badly on realistic tasks at a fraction of the same length. It is a narrow proxy widely read as evidence of a much broader capability.

Does long context replace RAG? No, though it changes what retrieval is for. Retrieval originally existed because models could not physically hold much text; it now exists because models cannot effectively use much text, which is a different constraint with a similar solution. Loading an entire corpus into a large window dilutes attention and adds distractors, so more context is not more usable knowledge past a point. The prevailing pattern is hybrid: retrieve the truly relevant material, which can still be substantial, then reason over that curated subset inside the long window. The window raises the ceiling on how much curated material you can use at once rather than removing the need to curate.

How can I get better results from long prompts? Treat context as scarce even when the window is not. Include less rather than more, since every irrelevant token dilutes attention and may act as a distractor. Place the most important material at the beginning or end of the input rather than the middle, where models underweight it. Be especially careful with content that closely resembles the answer without being it, because semantic similarity is what drives interference. Where possible, retrieve and curate rather than dumping whole documents. And measure where quality degrades on your own task and data instead of trusting either the advertised window size or a benchmark score.