Your RAG system isn't hallucinating. It never found the answer.

Here's a scene that plays out in a lot of teams.

You've built a RAG system over your company's documentation. It demos beautifully. Then real users arrive, and the answers get strange — vague, subtly wrong, occasionally confident about something that isn't in any document you own. Someone says the word "hallucination." The fix list gets written: try a better model, tighten the prompt, lower the temperature, add "only answer from the provided context" in bold.

None of it works. It doesn't work because the diagnosis was wrong.

The model didn't hallucinate. It answered faithfully — from material that didn't contain the answer, because your retrieval never found it.

Two halves, one blamed

RAG is two systems wearing one name. The retrieval half searches your documents and picks passages. The generation half reads those passages and writes an answer.

Only one of those halves is glamorous. All the demos, all the model announcements, all the conversation is about generation. Retrieval is a search engine with a new coat of paint, and nobody's excited about it.

But look at the failure. If retrieval hands the model three passages that don't contain the answer, what should a well-behaved model do? Say it doesn't know — which most will, some of the time. Or produce something plausible from adjacent material, which is what actually happens when the passages are nearly relevant. That output looks exactly like a hallucination. It has the same shape: confident, fluent, wrong. And it's not a generation failure at all.

The hallucination diagnosis is seductive because it's a known phenomenon with a known name. That's precisely why it gets over-applied. You can't fix retrieval by adjusting the model, and every hour spent on the model is an hour the actual bug survives.

Test the halves separately

The single most useful thing you can do to a RAG system costs an afternoon and no money.

Take thirty real questions — real ones, from actual users or realistic ones from someone who knows the domain. For each, note which document actually contains the answer. You now have a retrieval test set.

Run just the retrieval. Not the model. Look at the passages that come back and ask one question: is the answer in here?

That's it. That's the test. And it's the test almost nobody runs, because retrieval feels like plumbing and plumbing feels like it works.

The number that comes back tends to be sobering. If retrieval finds the answer 60% of the time, then your system has a hard ceiling at 60%, and no model on the market moves it. Swapping to a better model when retrieval is at 60% buys you a marginally better-written wrong answer.

If retrieval is at 95% and answers are still bad — now you have a generation problem, and now the fix list makes sense.

Why retrieval fails

Once you're looking at retrieval, the failures are specific and mostly fixable.

Chunking cut the answer in half. Documents get split into pieces before they're indexed, usually at some fixed size. If the answer spans a boundary, no chunk contains it, and no search can find what isn't there. A table split from its header. A policy split from its exception. This is the most common and most invisible failure, because everything looks fine in the code.

Semantic similarity isn't relevance. Embeddings find passages that are about the same topic. That's not the same as containing the answer. A question about your refund window will happily retrieve five passages that discuss refunds warmly and specify nothing. They're similar. They're useless. The system did what it was designed to do.

The vocabulary doesn't match. Users ask in their words; documents are written in yours. "Can I get my money back" and "reimbursement eligibility criteria" are the same question and are not close in embedding space. This is the case where old-fashioned keyword search often beats semantic search outright, and where combining both beats either.

One embedding model for everything. General-purpose embeddings were trained on general text. If your domain has its own language — clinical, legal, industrial, internal jargon — similarity in that space may not track similarity in yours.

Too many passages. This one's counter-intuitive, so it survives longest. Retrieving twenty documents instead of three feels safer — surely more context helps? It usually doesn't. Models attend unevenly across long inputs, and a correct passage buried in the middle of a large context can be effectively invisible. You've retrieved the answer and hidden it.

The fixes, in the order that pays

Fix retrieval before you touch anything else, and do it in this order, because the cheap ones fix most of it.

1. Look at your chunks. Actually read fifty of them. Not the code that makes them — the chunks themselves. You'll find headers orphaned from tables, sentences cut mid-clause, boilerplate repeated in every chunk so everything looks similar to everything. This is the highest-yield hour in the whole project and almost nobody spends it.

2. Add keyword search alongside semantic. Hybrid search — combining the two — is close to free and reliably beats either alone. Semantic catches meaning; keyword catches the exact product name, error code, or policy number that embeddings smear into approximate neighbours.

3. Rerank. Retrieve a wider net, then use a reranking model to order them properly and keep the top few. Rerankers are cheap, fast, and consistently the best value in the pipeline. This is where the "retrieve fewer, better passages" discipline actually gets implemented.

4. Fix the vocabulary gap. If users and documents speak differently, close it — expand queries, add synonyms, or generate the questions each document answers and index those instead.

5. Then consider a domain embedding model. This is real work and it helps, and it's fifth for a reason. Most teams that reach for it first would have got more from step one.

What to measure, once you're measuring

The thirty-question test gives you a yes/no per question, which is enough to know whether you have a problem. If you want to track improvement, two numbers do most of the work.

Recall@k — of the questions where the answer exists in your corpus, how often is it in the top k passages you retrieved? This is the ceiling on your whole system. Nothing downstream can exceed it. If recall@5 is 70%, then 30% of your users are getting an answer written from material that doesn't contain their answer, and the model's quality is irrelevant to them.

Precision — of the passages you retrieved, how many were actually useful? This one matters more than people expect, because of the crowding problem. Low precision means you're filling the context with near-misses, and near-misses are worse than nothing: they're similar enough to produce a confident wrong answer rather than an honest "I don't know."

Track recall first. It's the ceiling, it's the number that moves when you fix chunking, and it's the one that tells you whether to keep working on retrieval or move on.

One caution: measure on real questions. Questions written by the person who built the system are unconsciously shaped to the system's vocabulary, and they'll retrieve beautifully while real users fail. If you don't have real questions yet, get someone who's never seen the documents to write them.

What "grounded" actually requires

There's a deeper point under all this, and it's the reason RAG was worth building in the first place.

Retrieval is what gives an answer a source. The passage that produced it is right there — you can show it to a user, cite it, let them check. That property is why RAG beats fine-tuning for anything factual: not because it's more accurate in the abstract, but because it can be audited.

That only works if the passage genuinely contains the answer. A system that cites a source that doesn't support the claim is worse than one that cites nothing, because it manufactures the appearance of grounding. The citation becomes decoration — and users trust decorated answers more, not less.

So retrieval quality isn't a performance detail. It's the thing that makes the whole architecture honest.

When retrieval isn't the problem

Fairness requires the other side of this, because "it's always retrieval" would be its own lazy diagnosis.

If recall is high and answers are still poor, you have a real generation problem, and there are three shapes it usually takes.

The passage is there and the model ignored it. Often a crowding problem — you retrieved eight passages and the right one sits fourth, in the middle of a long context where attention is thinnest. Retrieve fewer. Rerank so the best one leads.

The model contradicts the passage. Rare, and genuinely a model failure when it happens. It's also the one case where a stronger model actually helps, which is why people reach for that fix so often — it works occasionally, which is enough to keep the habit alive.

The question needs synthesis across documents. The answer isn't in any single passage; it requires combining three. Standard RAG is poor at this by construction — it retrieves passages, not conclusions. That's an architecture mismatch, not a bug, and no amount of retrieval tuning fixes it.

Notice that two of the three are still not solved by a better model. That ratio is roughly the point of this whole piece.

The uncomfortable summary

If your RAG system is disappointing, the odds strongly favour retrieval as the culprit. And retrieval is the half nobody instruments, nobody evaluates, and nobody demos.

That's not a coincidence. The model is the exciting part, so attention goes there. The search engine is boring, so it's assumed to work. Meanwhile the actual quality of your product is being set by a chunking parameter someone picked in week one and never revisited.

Run the thirty-question test. It takes an afternoon and it will tell you, definitively, which half of your system is broken. Almost everything else you might do to a RAG pipeline is guessing until you have that number.

The concepts behind this: RAG, embeddings, vector databases, and hallucination — each explained at five levels from plain English to the research frontier.