Retrieval-Augmented Generation (RAG)
Letting an AI answer from a specific set of documents by looking them up as it responds — instead of relying only on what it memorized.
When not to use it
- When the knowledge is small and stable. If it fits comfortably in the prompt, put it in the prompt. RAG adds a retrieval system, a vector store, chunking decisions, and a whole new class of bugs.
- When you need the model to behave differently, not know more. RAG supplies facts; it doesn't change tone, format, or skill. That's a fine-tuning or prompting job.
- When retrieval quality will be poor. RAG on a messy, contradictory document pile produces confidently wrong answers with citations attached, which is worse than no answer at all.
Reach for something else instead
- A longer prompt. Context windows are large now. If your knowledge base is a handful of documents, skip the infrastructure.
- Fine-tuning when you need consistent style, format, or a narrow skill baked in — not fresh facts.
- Plain search with a human reading the results. Sometimes the honest answer is that people want the source document, not a paraphrase of it.
Sources & further reading
- Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the paper that named it.
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts — evidence that stuffing the context window is not the same as the model using it.
- Karpukhin et al. (2020), Dense Passage Retrieval for Open-Domain Question Answering — the retrieval half, which is where most RAG systems actually fail.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming retrieval fixes hallucination. It reduces it. The model can still ignore, misread, or blend the retrieved text.
- Chunking badly and blaming the model. Most disappointing RAG systems are retrieval failures wearing a generation costume — the right passage was never fetched.
- Skipping evaluation of the retrieval step on its own. Measure whether the right chunk comes back before you judge the answer.
At a glance
FieldLanguage & LLMs
Core idearetrieve-then-read
Addscurrent & private knowledge, citations
Fails whenretrieval fetches the wrong passage
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
RAG vs. Fine-tuning — adding knowledge at answer time vs. changing the model itself.