Context Window
The maximum amount of text an AI can consider at once — its short-term working memory, measured in tokens.
When not to use it
- As a substitute for retrieval. Pasting a whole knowledge base into a huge window is expensive on every call and often less accurate than fetching the right three paragraphs.
- Filling it because it's there. Models attend unevenly across long contexts — material buried in the middle gets used less reliably than material at either end.
Reach for something else instead
- RAG when the source material is bigger than the window, or changes, or only a fraction is relevant.
- Summarise-then-reason — compress earlier turns rather than resending everything.
- Caching repeated context, so the same preamble isn't billed on every request.
Sources & further reading
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts — models use the beginning and end more reliably than the middle.
- Press, Smith & Lewis (2021), Train Short, Test Long (ALiBi) — one of the position-encoding tricks that made longer contexts feasible.
- Dao et al. (2022), FlashAttention — why long contexts got cheaper without changing the maths.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Forgetting the answer shares the window. A prompt that nearly fills the context leaves no room to respond, and output gets truncated.
- Assuming a bigger window means better recall over that window. Longer context reliably costs more; it does not reliably work better.
- Measuring context in words or characters. It's tokens, and the conversion is not what you think for code, numbers, or non-English text.
At a glance
FieldLanguage & LLMs
Core ideathe model's working memory
Measured intokens
Key limitsshared input/output, cost, "lost in the middle"
DifficultyBeginner → Intermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4