Your RAG system isn't hallucinating. It never found the answer.

Here's a scene that plays out in a lot of teams.

You've built a RAG system over your company's documentation. It demos neatly. Then real users arrive, and the answers get strange, vague, subtly wrong, occasionally confident about something that isn't in any document you own. Someone says the word "hallucination." The fix list gets written: try a better model, tighten the prompt, lower the temperature, add "only answer from the provided context" in bold.

None of it works. It doesn't work because the diagnosis was wrong.

The model didn't hallucinate. It answered faithfully, from material that didn't contain the answer, because your retrieval never found it.

Two halves, one blamed

RAG is two systems wearing one name. The retrieval half searches your documents and picks passages. The generation half reads those passages and writes an answer.

Only one of those halves is glamorous. All the demos, all the model announcements, all the conversation is about generation. Retrieval is a search engine with a new coat of paint, and nobody's excited about it.

But look at the failure. If retrieval hands the model three passages that don't contain the answer, what should a well-behaved model do? Say it doesn't know, which most will, some of the time. Or produce something plausible from adjacent material, which is what actually happens when the passages are nearly relevant. That output looks exactly like a hallucination. It has the same shape: confident, fluent, wrong. And it's not a generation failure at all.

The hallucination diagnosis is seductive because it's a known phenomenon with a known name. That's precisely why it gets over-applied. You can't fix retrieval by adjusting the model, and every hour spent on the model is an hour the actual bug survives.

Test the halves separately

The single most useful thing you can do to a RAG system costs an afternoon and no money, and calibrating your similarity threshold is a close second.

Take thirty real questions, real ones, from actual users or realistic ones from someone who knows the domain. For each, note which document actually contains the answer. You now have a retrieval test set.

Run just the retrieval. Not the model. Look at the passages that come back and ask one question: is the answer in here?

That's it. That's the test. And it's the test almost nobody runs, because retrieval feels like plumbing and plumbing feels like it works.

The number that comes back tends to be sobering. If retrieval finds the answer 60% of the time, then your system has a hard ceiling at 60%, and no model on the market moves it. Swapping to a better model when retrieval is at 60% buys you a marginally better-written wrong answer.

If retrieval is at 95% and answers are still bad, now you have a generation problem, and now the fix list makes sense.

Why retrieval fails

Once you're looking at retrieval, the failures are specific and mostly fixable.

chunking cut the answer in half. Documents get split into pieces before they're indexed, usually at some fixed size. If the answer spans a boundary, no chunk contains it, and no search can find what isn't there. A table split from its header. A policy split from its exception. This is the most common and most invisible failure, because everything looks fine in the code.

Semantic similarity isn't relevance. Embeddings find passages that are about the same topic. That's not the same as containing the answer. A question about your refund window will happily retrieve five passages that discuss refunds warmly and specify nothing. They're similar. They're useless. The system did what it was designed to do.

The vocabulary doesn't match. Users ask in their words; documents are written in yours. "Can I get my money back" and "reimbursement eligibility criteria" are the same question and are not close in embedding space. This is the case where old-fashioned keyword search often beats semantic search outright, and where combining both beats either.

One embedding model for everything. General-purpose embeddings were trained on general text. If your domain has its own language, clinical, legal, industrial, internal jargon, similarity in that space may not track similarity in yours.

Too many passages. This one's counter-intuitive, so it survives longest. Retrieving twenty documents instead of three feels safer, surely more context helps? It usually doesn't. Models attend unevenly across long inputs, and a correct passage buried in the middle of a large context can be effectively invisible. You've retrieved the answer and hidden it.

The fixes, in the order that pays

Fix retrieval before you touch anything else, and do it in this order, because the cheap ones fix most of it.

1. Look at your chunks. Actually read fifty of them. Not the code that makes them, the chunks themselves. You'll find headers orphaned from tables, sentences cut mid-clause, boilerplate repeated in every chunk so everything looks similar to everything. This is the highest-yield hour in the whole project and almost nobody spends it.

2. Add keyword search alongside semantic. Hybrid search, combining the two, is close to free and reliably beats either alone. Semantic catches meaning; keyword catches the exact product name, error code, or policy number that embeddings smear into approximate neighbours.

3. Rerank. Retrieve a wider net, then use a reranking model to order them properly and keep the top few. Rerankers are cheap, fast, and consistently the best value in the pipeline. This is where the "retrieve fewer, better passages" discipline actually gets implemented.

4. Fix the vocabulary gap. If users and documents speak differently, close it, expand queries, add synonyms, or generate the questions each document answers and index those instead.

5. Then consider a domain embedding model. This is real work and it helps, and it's fifth for a reason. Most teams that reach for it first would have got more from step one.

What to measure, once you're measuring

The thirty-question test gives you a yes/no per question, which is enough to know whether you have a problem. If you want to track improvement, two numbers do most of the work.

Recall@k, of the questions where the answer exists in your corpus, how often is it in the top k passages you retrieved? This is the ceiling on your whole system. Nothing downstream can exceed it. If recall@5 is 70%, then 30% of your users are getting an answer written from material that doesn't contain their answer, and the model's quality is irrelevant to them.

Precision, of the passages you retrieved, how many were actually useful? This one matters more than people expect, because of the crowding problem. Low precision means you're filling the context with near-misses, and near-misses are worse than nothing: they're similar enough to produce a confident wrong answer rather than an honest "I don't know."

Track recall first. It's the ceiling, it's the number that moves when you fix chunking, and it's the one that tells you whether to keep working on retrieval or move on.

One caution: measure on real questions. Questions written by the person who built the system are unconsciously shaped to the system's vocabulary, and they'll retrieve neatly while real users fail. If you don't have real questions yet, get someone who's never seen the documents to write them.

What "grounded" actually requires

There's a deeper point under all this, and it's the reason RAG was worth building in the first place.

Retrieval is what gives an answer a source. The passage that produced it is right there. You can show it to a user, cite it, let them check. That property is why RAG beats fine-tuning for anything factual: not because it's more accurate in the abstract, but because it can be audited.

That only works if the passage contains the answer. A system that cites a source that doesn't support the claim is worse than one that cites nothing, because it manufactures the appearance of grounding. The citation becomes decoration, and users trust decorated answers more, not less.

So retrieval quality isn't a performance detail. It's the thing that makes the whole architecture honest.

The failure that precedes retrieval

Before the retriever runs, something has already decided what it is searching over, and that decision causes more failures than the retrieval algorithm does.

Chunking is a lossy choice made once. Split by character count and you cut sentences in half. Split by paragraph and you get chunks too varied in size for consistent embedding. Split by section and a long section becomes a chunk too coarse to match a specific question. There is no correct answer, only a set of trade-offs, and the one you picked is silently shaping every result the system will ever return.

Context is stripped at the boundary. A chunk that says "this applies only in the first case" is useless without the paragraph naming the cases. Retrieval will happily return it, the model will read it, and the answer will be confidently wrong in a way that traces back to a split point rather than to anything either component did badly.

Tables and structure do not survive. Most chunking flattens a table into a run of text where the relationship between a value and its column header is gone. If your corpus is largely tabular, this is likely your dominant failure and no amount of retrieval tuning addresses it.

The practical implication: when retrieval underperforms, look at the chunks it returned before looking at the retriever. Frequently the right chunk was found and was not usable.

What good looks like operationally

A retrieval system that stays good is not one that was configured well once.

Log what was retrieved, always. Not just the answer, the passages. Without this you cannot distinguish a retrieval failure from a generation failure after the fact, which means every investigation starts by trying to reproduce the problem.

Sample and read them. Twenty retrievals a week, read by a person who knows the domain. This finds systematic problems that aggregate metrics hide, particularly whole categories of question that never retrieve well because the corpus does not cover them and nobody noticed.

Track the abstention rate. If the system is permitted to say it does not know, how often does it? A rate of zero means the permission is not real, and every question is being answered from whatever came back regardless of relevance.

Re-evaluate when the corpus changes. Adding documents changes what competes for retrieval. A system that worked at ten thousand chunks can degrade at a hundred thousand without any code change, because the neighbourhood around each query got more crowded.

When retrieval isn't the problem

Fairness requires the other side of this, because "it's always retrieval" would be its own lazy diagnosis.

If recall is high and answers are still poor. You have a real generation problem, and there are three shapes it usually takes.

The passage is there and the model ignored it. Often a crowding problem, you retrieved eight passages and the right one sits fourth, in the middle of a long context where attention is thinnest. Retrieve fewer. Rerank so the best one leads.

The model contradicts the passage. Rare, and a model failure when it happens. It's also the one case where a stronger model actually helps, which is why people reach for that fix so often. It works occasionally, which is enough to keep the habit alive.

The question needs synthesis across documents. The answer isn't in any single passage; it requires combining three. Standard RAG is poor at this by construction, it retrieves passages, not conclusions. That's an architecture mismatch, not a bug, and no amount of retrieval tuning fixes it.

Notice that two of the three are still not solved by a better model. That ratio is roughly the point of this whole piece.

The uncomfortable summary

If your RAG system is disappointing, the odds strongly favour retrieval as the culprit. And retrieval is the half nobody instruments, nobody evaluates, and nobody demos.

That's not a coincidence. The model is the exciting part, so attention goes there. The search engine is boring, so it's assumed to work. Meanwhile the actual quality of your product is being set by a chunking parameter someone picked in week one and never revisited.

Run the thirty-question test. It takes an afternoon and it will tell you, definitively, which half of your system is broken. Almost everything else you might do to a RAG pipeline is guessing until you have that number.

The concepts behind this: RAG, embeddings, vector databases, and hallucination, each explained at five levels from plain English to the research frontier.

The short version

In a RAG system, the model gets most of the blame for wrong answers, but the fault usually lies in retrieval. A language model can only answer from what is placed in its context, so if the right information was never fetched, or was buried among distracting near-matches, or was split badly during chunking, the model produces a poor answer no matter how capable it is. This is why upgrading the model rarely fixes a struggling RAG system, while improving chunking, search, reranking, and filtering usually does. The discipline is to measure retrieval directly, checking whether the correct chunk was fetched, rather than judging only the final answer. A RAG system's answers are only as good as what it retrieves, so when they are wrong, look at retrieval before blaming the model.

Common questions

Why does my RAG system give wrong answers even with a good model? Usually because the right passage was never retrieved. When the retrieval step misses, the model answers from its parameters instead of your documents, which looks like hallucination but is a search failure. Fix retrieval before blaming the model.

What does a "grounded" answer actually require? That the claim traces to a retrieved passage that supports it. Grounding isn't a prompt instruction ("only answer from context"); it's a property you have to retrieve for and then verify, because the model will fill gaps confidently if the passage is missing.

When is retrieval not the problem? When the right passage was retrieved and the model still got it wrong, a reasoning or synthesis failure, or when the answer simply isn't in your corpus. Diagnosing which case you're in is the difference between fixing search and fixing the model.

What is retrieval in RAG? Retrieval is the step where a RAG system finds the pieces of your knowledge base most relevant to a question and supplies them to the model. Typically the question is turned into an embedding and matched against stored document chunks by similarity, sometimes combined with keyword search and reranking, returning the top few chunks. Those chunks become the context the model answers from. Retrieval is the part of RAG that determines whether the model even has the information it needs, which is why its quality, not the model's, usually decides whether a RAG answer is right.

Why do most RAG failures come from retrieval, not the model? Because a language model can only answer well from the information placed in its context, and if retrieval fails to fetch the right material, no model capability can recover what is not there. When a RAG system gives a wrong or vague answer, the usual cause is that the relevant chunk was never retrieved, was retrieved alongside distracting near-matches, or was split badly during chunking so the key fact was severed from its context. The model then does its best with an inadequate context. Fixing the model rarely helps; fixing what gets retrieved and how it is assembled usually does.

How do you improve a RAG system's retrieval? Work on the parts that determine what reaches the model. Improve chunking so each chunk is a coherent, self-contained unit rather than an arbitrary slice that severs facts from context. Improve the search itself, often by combining semantic and keyword matching, and add reranking to push the most relevant results to the top. Filter by metadata to narrow the search space. And measure retrieval directly, checking whether the right chunk was fetched, rather than only judging the final answer, so you can tell retrieval failures from generation ones and fix the actual bottleneck.

Can a better language model fix a RAG system's wrong answers? Usually not, if the problem is retrieval, which it most often is. If the correct information was never fetched into the model's context, a more capable model has nothing better to work with and produces the same kind of wrong or vague answer, sometimes more convincingly. A stronger model helps only when the right context is present but the model reasons over it poorly, which is the rarer case. Before upgrading the model, check whether retrieval actually surfaced the needed information; fixing chunking, search, and reranking typically does far more for answer quality.