RAG vs. Fine-tuning: which should you actually use?

When people first try to make a large language model useful for their own work, they hit the same fork in the road. The model is fluent and capable, but it doesn't know your documents, and it doesn't quite behave the way you want. Two techniques promise to fix that: retrieval-augmented generation and fine-tuning. They sound interchangeable. They are not, and picking the wrong one is one of the most common — and most expensive — early mistakes.

Here's the short version, and if you remember nothing else, remember this:

RAG is for knowledge. Fine-tuning is for behaviour.

What each one actually does

Retrieval-augmented generation, or RAG, hands the model an open book. When a question comes in, the system searches a collection of your documents, pulls out the most relevant passages, and gives them to the model along with the question. The model then answers from that material. Nothing about the model changes — you're just supplying it with the right pages at the right moment.

Fine-tuning does something fundamentally different. It continues the model's training on your own examples, so its behaviour shifts. After fine-tuning, a new tendency is baked into the model's weights — a consistent tone, a fixed output format, a specialised way of handling a task. You're not giving it a book to read; you're sending it on a training course.

That distinction — supplying knowledge at answer time versus changing the model itself — is the whole decision.

The rule of thumb

Reach for RAG when the problem is that the model doesn't know something:

• It needs current information that changes over time.
• It needs to answer from private documents it never saw in training.
• You want answers that cite their sources.
• The underlying facts get updated often.

Reach for fine-tuning when the problem is that the model doesn't behave the way you need:

• You need a consistent voice or persona the model won't hold with prompting alone.
• You need a strict output structure every single time.
• You're teaching a narrow, repeatable skill the base model does inconsistently.

If your answer to "what's wrong?" is "it doesn't know X," that's RAG. If it's "it doesn't act like Y," that's fine-tuning.

Why the knowledge/behaviour split holds up

This isn't a mnemonic someone invented to simplify a lecture. It falls out of how each technique works.

Fine-tuning adjusts weights during training. Whatever it learns is fixed at the moment training ends. If you fine-tune a model on your product catalogue in March and the prices change in April, the model is now confidently wrong and there is no patch — only another training run. Worse, it can't tell you where an answer came from, because the answer isn't from anywhere in particular. It's a tendency distributed across billions of numbers.

Retrieval does its work at query time. The document is fetched when the question is asked, which means updating your knowledge is updating a file. Change the document, and the next answer changes. You also get provenance almost for free: the passage that produced the answer is right there, and you can show it to the user. In any setting where someone might reasonably ask "says who?" — legal, medical, financial, internal policy — that property alone decides it.

So the rule isn't a simplification. Facts move and need attribution; retrieval handles both. Behaviour should be stable and doesn't need a citation; training handles that.

The mistake almost everyone makes

The most common error is reaching for fine-tuning too early. It sounds more powerful and more permanent, so people assume it's the serious option — the thing real engineers do, as opposed to a workaround.

That instinct is backwards, and it's expensive. Fine-tuning is slow to set up, needs a dataset you probably don't have yet, is painful to update, and is the wrong tool for adding knowledge in the first place. Teams routinely spend six weeks assembling training data to teach a model facts that a retrieval pipeline would have served on day three — and then discover the facts have changed.

Meanwhile, the cheap options solve more than people expect. The honest sequence:

1. Write a better prompt. Be specific. Say what to do when the answer isn't available. This is free and takes an afternoon. 2. Add examples to the prompt. Two or three good demonstrations of the output you want will get you a surprising distance toward what people assume needs training. Skipping this step is the most common expensive mistake in the field. 3. Add retrieval if the model needs facts it doesn't have. 4. Then, and only then, consider fine-tuning — if a genuine behavioural gap remains after all of the above, and you can point at it.

Most projects should stop at step 2 or 3. The ones that reach step 4 have earned it, and they know exactly what they're buying.

When fine-tuning genuinely wins

None of this means fine-tuning is a trap. There are cases where it's clearly right, and they share a shape.

When prompting works but inconsistently. You've written good examples, and the model gets it right most of the time and drifts the rest. That inconsistency is exactly what training fixes.

When your prompt has become enormous. If you're sending 800 tokens of rules and examples on every single call, you're paying for that preamble forever. At volume, folding it into the weights is cheaper — and faster, since there's less to process. Do the arithmetic; the crossover point arrives sooner than people expect.

When you need a narrow skill done reliably. Classifying support tickets into your twelve categories, extracting your specific fields, matching your house style. Narrow, repeatable, well-defined — and a small fine-tuned model will often beat a large general one at exactly that job, for a fraction of the running cost.

If you do fine-tune, start with LoRA rather than full fine-tuning. It's cheaper, faster, fits on hardware you probably already have, and is good enough for nearly everything. And build a held-out evaluation set before you train, or "it feels better" will be the only evidence you have that an expensive process worked.

What each one actually costs

The comparison people run is accuracy. The comparison that decides projects is effort, and the two techniques are lopsided in a way the "vs." framing hides.

RAG costs you a pipeline. Documents have to be collected, split into chunks, embedded, and stored somewhere searchable. Then every query pays for a search plus a larger prompt, because you're sending retrieved passages along with the question. That's ongoing cost per request, forever, and it scales with how much context you stuff in. Against that: you can have something working in days, updating it is updating a file, and nobody needs to assemble a training set.

Fine-tuning costs you a dataset. That's the real bill, and it's paid in human hours, not compute. You need hundreds to thousands of examples of the behaviour you want, consistent enough to learn from — and the assembling is the project. The training run itself is often cheap and quick by comparison. Against that: once it's done, inference is cheaper than RAG, because your prompts are shorter and there's no retrieval step.

So the shape is: RAG is cheap to start and costs a little forever. Fine-tuning is expensive to start and then makes every request cheaper. At low volume RAG wins on arithmetic alone. At high volume with stable behaviour, fine-tuning eventually pays back. Most teams never reach that crossover, and the ones that do usually know it.

There's a third cost nobody budgets for: evaluation. Both techniques need a way to tell whether they worked, and neither provides one. A fine-tune without a held-out test set is a vibe. A RAG system without retrieval metrics is a mystery. That work is unglamorous and it's the difference between improving something and moving it around.

Where RAG actually goes wrong

RAG has its own failure mode, and it's badly misunderstood.

When a RAG system gives a disappointing answer, the instinct is to blame the model — it must have misread the passage, or made something up. Usually that's not what happened. Usually the right passage was never retrieved in the first place. The model answered faithfully from the wrong material, or from nothing much at all.

This matters because it changes what you fix. If you assume generation is broken, you'll try a bigger model, tweak the prompt, adjust temperature, and get nowhere. The problem is in the half of the system nobody demos: chunking, embeddings, and search quality.

So evaluate retrieval on its own, before you judge the answers. Take a set of real questions and check whether the passages coming back actually contain the answer. If they don't, no model on earth will save you. If they do, and the answers are still bad, then you have a generation problem.

The other quiet trap: stuffing more into the context window isn't the same as the model using it. Models attend unevenly across long inputs, and a passage buried in the middle of a large context can be effectively invisible. Retrieving twenty documents instead of three often makes answers worse, not better.

They're not rivals

The framing "RAG vs. fine-tuning" is useful for choosing, but in practice the two are often combined — and the better systems usually do combine them.

You might fine-tune a model to reliably produce your house tone and output format, and wrap it in RAG so it answers from your live documentation. One handles behaviour, the other handles knowledge. They sit on different axes, which is exactly why the "vs." is a little misleading once you understand them.

A support assistant shows the shape well. Fine-tuning teaches it to respond in your brand's voice, structure replies consistently, and never promise refunds it can't authorise. Retrieval gives it today's policies, this customer's order history, and the current status of their delivery. Neither technique could do the other's job, and the system needs both.

The question underneath

So the real question isn't "which is better." It's "what's actually broken — the model's knowledge, or its behaviour?"

That question is harder than it sounds, because "the AI isn't working" is where most teams start, and it isn't a diagnosis. Push on it. Is it saying things that are wrong, or saying things wrong? Those are different sentences, and different projects.

Answer that honestly, and the choice makes itself.

Want the underlying concepts in more depth? Read the full entries on RAG and fine-tuning, each explained at five levels from plain English to the research frontier. The prompt engineering entry covers the cheaper steps worth exhausting first.