Distillation
Training a small model to imitate a large one — which works better than training the small model directly, for reasons that are still argued about.
Reviewed July 11, 2026Contested
When not to use it
- When you need the teacher's breadth. Distillation narrows as it shrinks, and the narrowing is invisible on in-distribution benchmarks.
- When the capacity gap is large. A tiny student can't represent a huge teacher's function, and the result is worse than a smaller teacher would have produced.
- From an API you don't own. It's against most terms of service, and it's now an accusation with consequences.
- When quantization would do. If the problem is memory rather than architecture, quantizing is simpler and lossless-ish.
Reach for something else instead
- Quantization — smaller weights, same model, no retraining.
- Pruning — remove weights that don't matter.
- LoRA on a small base — if you want a specialist, adapting a small model directly may beat distilling a big one.
- Training the small model on more real data — sometimes wins, and it's the baseline people skip.
Read more on the blog
- Knowledge distillation: how small models learn from big onesThe small, fast AI models you run on a laptop or serve cheaply to millions of users were often not just shrunk from big models. They were taught by them. Knowledge distillation trains a compact student model to mimic a large teacher, and the surprising part is that the student learns more from the teacher's uncertainty than from the raw right answers.
- RAG vs fine-tuning: the decision, honestlyThe most common question in applied AI, answered without the hedging: when do you use retrieval, when do you fine-tune, when do you need both, and what almost every team gets wrong about the choice.
- What is fine-tuning? How to specialize an AI modelFine-tuning takes a general-purpose model and adapts it to your specific task by continuing its training on your examples. It is one of the three ways to shape a model, the most powerful and the most misused. Here is what it actually does, how LoRA made it cheap, and the honest answer to whether you should do it.
- What is synthetic data? Training AI on AI-made dataFaced with running out of human text to train on, AI labs increasingly train models on data the models generate themselves. This is synthetic data, and it comes with a famous warning called model collapse. The resolution of that tension, that verification and curation are what separate collapse from improvement, is the whole story and one of the most important ideas in how modern AI is built.
Further reading
- Hinton, Vinyals & Dean (2015), Distilling the Knowledge in a Neural Network — the paper, the temperature trick, and the dark-knowledge story.
- Buciluă et al. (2006), Model Compression — the original idea, nine years earlier and largely forgotten.
- Cho & Hariharan (2019), On the Efficacy of Knowledge Distillation — the awkward finding that better teachers don't reliably make better students.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Distilling at temperature 1. The dark knowledge is in the small probabilities and you've flattened them out of existence.
- Assuming the best teacher makes the best student. The evidence says there's a sweet spot in the size ratio.
- Evaluating only in-distribution. That's exactly where distillation looks best and hides what it lost.
- Forgetting the T² gradient scaling and wondering why the loss balance is wrong.
- Treating "train on GPT outputs" as legally settled. It isn't.
At a glance
FieldDeep Learning
Ideastudent copies teacher's full output distribution
Key tricktemperature, to expose small probabilities
Typical gain~10× cheaper for a few percent
Hidden costnarrows as it shrinks
Mechanismstill argued about
DifficultyIntermediate
Often compared with
Distillation vs. quantization — one trains a new smaller model that behaves like the old one; the other shrinks the same model's numbers. Quantization keeps the breadth; distillation may not.
Where this sits
6 concepts come first. Understanding it opens up 1 more.
5Levelsteps in
6Needs firstconcepts
1Opens up0% of 310
2Areascrosses fields
Computed from the prerequisite graph, not assigned. How this works