LoRA (Low-Rank Adaptation)
A way to fine-tune a huge model by training a tiny add-on instead of the model itself — cheap enough that one GPU will do, and good enough that it became the default.
Reviewed July 11, 2026Stable
When not to use it
- When the problem is knowledge, not behaviour. LoRA is weakest at teaching facts. If the model needs to know your documents, retrieval is the tool and no rank setting fixes that.
- When you haven't tried prompting. Few-shot examples cost nothing and are competitive on a surprising share of tasks. Fine-tuning before prompting is the most common expensive mistake in this area.
- When you have no evaluation set. Without one, "it feels better" will be your only evidence that the training worked, and it will be wrong about as often as it's right.
- When the task is genuinely far from pretraining. Rare, but real: if you need behaviour the base model has no foundation for, low-rank adaptation may not have the capacity and full fine-tuning is the honest answer.
Reach for something else instead
- Few-shot prompting — free, instant, no training. Always the first move.
- RAG — for anything where the answer lives in documents rather than in behaviour.
- Full fine-tuning — when LoRA has genuinely run out of capacity and you can prove it with an eval.
- Prompt caching — if the problem is that your prompt is long and expensive, this is cheaper than training.
Read more on the blog
- RAG vs fine-tuning: the decision, honestlyThe most common question in applied AI, answered without the hedging: when do you use retrieval, when do you fine-tune, when do you need both, and what almost every team gets wrong about the choice.
- What is fine-tuning? How to specialize an AI modelFine-tuning takes a general-purpose model and adapts it to your specific task by continuing its training on your examples. It is one of the three ways to shape a model, the most powerful and the most misused. Here is what it actually does, how LoRA made it cheap, and the honest answer to whether you should do it.
- How quantization shrinks AI models without breaking themA 70-billion-parameter model needs about 140 GB of memory at full precision. Your laptop has 16. Quantization is how the model fits anyway, by storing each weight in far fewer bits, and the surprising part is that you can throw away most of that precision and the model barely notices. Here is why, and where it finally breaks.
Further reading
- Hu et al. (2022), LoRA: Low-Rank Adaptation of Large Language Models — the original, and still the clearest statement of the idea.
- Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs — 4-bit base plus adapter; why single-GPU fine-tuning of large models became normal.
- Houlsby et al. (2019), Parameter-Efficient Transfer Learning for NLP — the adapter work LoRA descends from, and the inference-latency problem LoRA solves.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Using LoRA to teach facts. It's the wrong tool and the failure is quiet — the model gets the tone of knowing right while getting the content wrong.
- Cranking rank because more sounds better. Higher rank costs more and often does nothing; the eval decides, not intuition.
- Training on a few dozen examples and expecting transformation. LoRA needs less data than full fine-tuning, not no data.
- Forgetting to merge for production, then wondering about latency. Unmerged adapters add a forward-pass cost that merging removes entirely.
At a glance
FieldLanguage & LLMs
What it changesbehaviour, not knowledge
Typical rank8–16
Adapter sizemegabytes, vs. tens of gigabytes
Added latency once mergednone
DifficultyIntermediate
Often compared with
LoRA vs. full fine-tuning — the same job at a fraction of the cost for most tasks. The gap only shows up when the target behaviour is far from anything the base model learned.
Where this sits
A destination. 26 concepts lead here, and nothing in the corpus depends on it.
12Levelsteps in
26Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstFine-tuning
Computed from the prerequisite graph, not assigned. How this works