LoRA (Low-Rank Adaptation)
A way to fine-tune a huge model by training a tiny add-on instead of the model itself — cheap enough that one GPU will do, and good enough that it became the default.
When not to use it
- When the problem is knowledge, not behaviour. LoRA is weakest at teaching facts. If the model needs to know your documents, retrieval is the tool and no rank setting fixes that.
- When you haven't tried prompting. Few-shot examples cost nothing and are competitive on a surprising share of tasks. Fine-tuning before prompting is the most common expensive mistake in this area.
- When you have no evaluation set. Without one, "it feels better" will be your only evidence that the training worked, and it will be wrong about as often as it's right.
- When the task is genuinely far from pretraining. Rare, but real: if you need behaviour the base model has no foundation for, low-rank adaptation may not have the capacity and full fine-tuning is the honest answer.
Reach for something else instead
- Few-shot prompting — free, instant, no training. Always the first move.
- RAG — for anything where the answer lives in documents rather than in behaviour.
- Full fine-tuning — when LoRA has genuinely run out of capacity and you can prove it with an eval.
- Prompt caching — if the problem is that your prompt is long and expensive, this is cheaper than training.
Sources & further reading
- Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models — the original, and still the clearest statement of the idea.
- Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs — 4-bit base plus adapter; why single-GPU fine-tuning of large models became normal.
- Houlsby et al. (2019), Parameter-Efficient Transfer Learning for NLP — the adapter work LoRA descends from, and the inference-latency problem LoRA solves.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Using LoRA to teach facts. It's the wrong tool and the failure is quiet — the model gets the tone of knowing right while getting the content wrong.
- Cranking rank because more sounds better. Higher rank costs more and often does nothing; the eval decides, not intuition.
- Training on a few dozen examples and expecting transformation. LoRA needs less data than full fine-tuning, not no data.
- Forgetting to merge for production, then wondering about latency. Unmerged adapters add a forward-pass cost that merging removes entirely.
At a glance
FieldLanguage & LLMs
What it changesbehaviour, not knowledge
Typical rank8–16
Adapter sizemegabytes, vs. tens of gigabytes
Added latency once mergednone
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
LoRA vs. full fine-tuning — the same job at a fraction of the cost for most tasks. The gap only shows up when the target behaviour is far from anything the base model learned.