Chunking
Cutting documents into retrievable pieces — the least glamorous decision in RAG, and the one that most often decides whether it works.
When not to use it
- When the documents fit in the context window and don't change. Then you don't need retrieval at all, and chunking is machinery in service of a problem you don't have.
- When the documents have hard structural units already. Product records, ticket entries, FAQ pairs — the unit exists. Cutting it into arbitrary pieces destroys the thing you were given.
- As a tuning exercise before you've read the output. Adjusting
chunk_sizewithout reading chunks is guessing with extra steps.
Reach for something else instead
- Structural splitting — use the document's own headings and sections. Usually better than any size-based rule.
- Small-to-big retrieval — retrieve precise, return contextual. Sidesteps the size trade-off.
- Contextual retrieval — prepend document context to each chunk before embedding.
- Full-document context — for small, stable corpora, skip the whole apparatus.
Sources & further reading
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts — why "just retrieve more and let the model sort it out" underperforms.
- Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the RAG paper; the retrieval unit is a design decision from the start.
- Karpukhin et al. (2020), Dense Passage Retrieval for Open-Domain Question Answering — passage-level retrieval and why the passage boundary matters.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Never reading the chunks. The highest-yield hour in the project, routinely skipped.
- Fixed-size splitting on structured documents, cutting tables from their headers and clauses from their conditions.
- No overlap, so any answer sitting on a boundary is unretrievable and you never find out.
- Treating chunk size as a global constant. It's a property of your documents and your questions, not of the field.
- Tuning the embedding model while the chunking is broken. You're polishing the search over material that doesn't contain the answer.
At a glance
FieldLanguage & LLMs
Where it sitsbefore embedding, upstream of everything
Typical overlap10–20%
Best defaultstructural, then recursive
Failure modesilent
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Chunking vs. long context — cutting documents up vs. putting them in whole. Long context doesn't remove the problem; it moves it to which documents, which is retrieval again.