Chunking
Cutting documents into retrievable pieces — the least glamorous decision in RAG, and the one that most often decides whether it works.
Reviewed July 11, 2026Stable
When not to use it
- When the documents fit in the context window and don't change. Then you don't need retrieval at all, and chunking is machinery in service of a problem you don't have.
- When the documents have hard structural units already. Product records, ticket entries, FAQ pairs — the unit exists. Cutting it into arbitrary pieces destroys the thing you were given.
- As a tuning exercise before you've read the output. Adjusting
chunk_sizewithout reading chunks is guessing with extra steps.
Reach for something else instead
- Structural splitting — use the document's own headings and sections. Usually better than any size-based rule.
- Small-to-big retrieval — retrieve precise, return contextual. Sidesteps the size trade-off.
- Contextual retrieval — prepend document context to each chunk before embedding.
- Full-document context — for small, stable corpora, skip the whole apparatus.
Read more on the blog
- Beyond vector search: how RAG actually works in 2026RAG stopped being "vector database plus a language model" a while ago. Here's how retrieval-augmented generation actually works now, chunking, embeddings, reranking, knowledge graphs, and agentic retrieval, and where each piece quietly breaks.
- Why long context windows fail: the lost-in-the-middle problemModels now advertise context windows of a million tokens, and they pass the standard needle-in-a-haystack test almost perfectly. Yet their accuracy on real long-context work collapses far below those advertised limits. The gap between what a model will accept and what it can actually use is one of the most practically important facts about modern AI.
- Your RAG system isn't hallucinating. It never found the answer.When a RAG system gives a bad answer, almost everyone blames the model. Usually the right passage was never retrieved, and that changes everything about how you fix it.
- Context engineering: the skill that replaced prompt engineeringPrompt engineering didn't die, it got absorbed. The bottleneck moved from how you phrase a request to what information surrounds it. Here's what context engineering actually is, why it took over in 2026, and the discipline underneath the buzzword.
Further reading
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts — why "just retrieve more and let the model sort it out" underperforms.
- Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the RAG paper; the retrieval unit is a design decision from the start.
- Karpukhin et al. (2020), Dense Passage Retrieval for Open-Domain Question Answering — passage-level retrieval and why the passage boundary matters.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Never reading the chunks. The highest-yield hour in the project, routinely skipped.
- Fixed-size splitting on structured documents, cutting tables from their headers and clauses from their conditions.
- No overlap, so any answer sitting on a boundary is unretrievable and you never find out.
- Treating chunk size as a global constant. It's a property of your documents and your questions, not of the field.
- Tuning the embedding model while the chunking is broken. You're polishing the search over material that doesn't contain the answer.
At a glance
FieldLanguage & LLMs
Where it sitsbefore embedding, upstream of everything
Typical overlap10–20%
Best defaultstructural, then recursive
Failure modesilent
DifficultyBeginner
Often compared with
Chunking vs. long context — cutting documents up vs. putting them in whole. Long context doesn't remove the problem; it moves it to which documents, which is retrieval again.
Where this sits
2 concepts come first. Understanding it opens up 5 more.
2Levelsteps in
2Needs firstconcepts
5Opens up2% of 310
2Areascrosses fields
Learn these firstTokenization
Computed from the prerequisite graph, not assigned. How this works