Needle in a Haystack
The test that hides a fact in a long document and asks the model to find it — and the reason a model can pass it at 128k tokens and still be useless at 32k.
Reviewed July 16, 2026Stable
When not to use it
- As evidence that long context works. It tests retrieval of a distinctive string, which is the easiest thing a long window can do.
- As a vendor comparison. Everyone passes it; the chart discriminates between nobody.
- Instead of your own eval. Your documents don't contain a conveniently out-of-place sentence, and your questions don't have verbatim answers.
Reach for something else instead
- RULER or another multi-hop long-context suite tests what you actually care about.
- Your own documents and questions are the only eval that answers your question, and they take an afternoon.
- RAG frequently outperforms long context on the same task at a fraction of the cost — worth testing before paying for the window.
Sources & further reading
- Hsieh et al. (2024), RULER: What's the Real Context Size of Your Long-Context Language Models? — the multi-hop, multi-needle successor; claimed length substantially exceeds effective length.
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts — uneven access across the window; the mechanism behind the position effect.
- Kamradt (2023), LLMTest_NeedleInAHaystack — the original open-source harness the industry standardised on.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading a green needle chart as long-context competence. It measures string-spotting; the model didn't have to read the haystack.
- Confusing claimed context with effective context. RULER found nearly all models degrade well before their advertised length.
- Testing single-needle only. Multi-hop and aggregation are where models fail, and where real work lives.
At a glance
FieldLanguage & LLMs
Testsretrieval of one distinctive sentence
Weaknessclose to string matching
Better successorRULER (2024)
DifficultyBeginner