Distillation
Training a small model to imitate a large one — which works better than training the small model directly, for reasons that are still argued about.
When not to use it
- When you need the teacher's breadth. Distillation narrows as it shrinks, and the narrowing is invisible on in-distribution benchmarks.
- When the capacity gap is large. A tiny student can't represent a huge teacher's function, and the result is worse than a smaller teacher would have produced.
- From an API you don't own. It's against most terms of service, and it's now an accusation with consequences.
- When quantization would do. If the problem is memory rather than architecture, quantizing is simpler and lossless-ish.
Reach for something else instead
- Quantization — smaller weights, same model, no retraining.
- Pruning — remove weights that don't matter.
- LoRA on a small base — if you want a specialist, adapting a small model directly may beat distilling a big one.
- Training the small model on more real data — sometimes wins, and it's the baseline people skip.
Sources & further reading
- Hinton, Vinyals & Dean (2015), Distilling the Knowledge in a Neural Network — the paper, the temperature trick, and the dark-knowledge story.
- Buciluă et al. (2006), Model Compression — the original idea, nine years earlier and largely forgotten.
- Cho & Hariharan (2019), On the Efficacy of Knowledge Distillation — the awkward finding that better teachers don't reliably make better students.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Distilling at temperature 1. The dark knowledge is in the small probabilities and you've flattened them out of existence.
- Assuming the best teacher makes the best student. The evidence says there's a sweet spot in the size ratio.
- Evaluating only in-distribution. That's exactly where distillation looks best and hides what it lost.
- Forgetting the T² gradient scaling and wondering why the loss balance is wrong.
- Treating "train on GPT outputs" as legally settled. It isn't.
At a glance
FieldDeep Learning
Ideastudent copies teacher's full output distribution
Key tricktemperature, to expose small probabilities
Typical gain~10× cheaper for a few percent
Hidden costnarrows as it shrinks
Mechanismstill argued about
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Distillation vs. quantization — one trains a new smaller model that behaves like the old one; the other shrinks the same model's numbers. Quantization keeps the breadth; distillation may not.