Knowledge distillation: how small models learn from big ones

The capable AI models that are small enough to run on a laptop, cheap enough to serve to millions of users, or fast enough for a phone were, in many cases, not simply shrunk-down versions of big models. They were taught by them. The technique is called knowledge distillation, and it trains a small student model to mimic the behaviour of a large, expensive teacher model, capturing much of the teacher's capability at a fraction of the size and cost. It is one of the most important and least understood techniques in modern AI, because it is a large part of how frontier capability gets compressed into something you can actually afford to run.

The counterintuitive part, and the reason distillation works as well as it does, is that a student learns better from the teacher's uncertainty than from the plain correct answers. This piece explains what distillation is, the idea of dark knowledge that makes it effective, why a small model taught by a big one can outperform the same small model trained on raw data, how it differs from the other ways of shrinking a model, and why it has become standard practice for building deployable AI. It is the final step in the story of how a model goes from a frontier research artifact to something running on the device in your hand.

The setup: teacher and student

Distillation involves two models. The teacher is large, capable, and expensive: a frontier-scale model that performs well but costs too much to run at scale. The student is small, fast, and cheap: the model you actually want to deploy. Normally you would train the student directly on your data, but a small model trained from scratch tends to reach only modest performance, because it lacks the capacity to discover good solutions on its own. Distillation takes a different route: instead of training the student on the raw data alone, you train it to imitate the teacher's outputs.

The result, when it works, is a student that captures a surprising amount of the teacher's capability while being far smaller. Typical distilled models deliver large cost reductions, often several times to tens of times cheaper to run, with only a small loss in quality. That trade is why distillation moved from a research curiosity to a standard production step: it is how a lab takes a model too heavy to serve and produces a version light enough to ship.

Dark knowledge: why the teacher's uncertainty is the point

The heart of distillation, and the idea worth understanding even if you remember nothing else, is what the student learns from. When you train a model normally, you use hard labels: an image is labelled 100 percent cat, and everything else is zero. That label is correct but impoverished. It tells the model the answer and nothing about how the answer relates to the alternatives.

A trained teacher produces something richer: a soft label, a full probability distribution over the possibilities. Shown a picture of a cat, the teacher might output 85 percent cat, 10 percent dog, and 5 percent car. Look at what that distribution encodes. The teacher is saying not just that this is a cat, but that it is somewhat dog-like and much less car-like, that a cat sits closer to a dog than to a vehicle in the space of things. That relational information, the structure of how the teacher sees the possibilities, is present in the soft distribution and completely absent from the hard label. This extra signal has a name: dark knowledge, the information hidden in a teacher's soft predictions about how classes relate to one another.

When the student trains on these soft labels, it learns more than the right answer. It learns a shadow of the teacher's whole way of organising the problem, which classes are similar, which distinctions are subtle, how confident to be. This is why a student distilled from a teacher generalises better than the identical small model trained on the raw hard labels: the teacher's uncertainty is not noise to be discarded, it is a compressed lesson about the structure of the task, and the student that absorbs it inherits some of the teacher's judgment rather than only its conclusions.

Temperature: turning up the dark knowledge

There is a practical wrinkle worth knowing, because it connects to another familiar idea. A well-trained teacher is often very confident, so its soft label might be 99 percent cat and almost nothing else, which hides the relational information you want the student to see. To fix this, distillation applies a temperature to the teacher's outputs, the same mechanism that controls randomness in text generation, but used here for a different purpose. Dividing the teacher's raw scores by a temperature before converting them to probabilities softens the distribution, flattening that 99 percent down so the small but meaningful probabilities on the other classes become visible. A higher temperature exposes more of the dark knowledge, the subtle relationships between options, giving the student a richer signal to learn from. The same knob that makes a chatbot more creative is, in distillation, the dial that controls how much of the teacher's nuanced understanding gets transmitted.

Why a small student can punch above its weight

The striking consequence is that distillation can let a small model reach performance it could never find on its own. A compact model trained directly on data is limited by its own capacity to search for good solutions from scratch. But a compact model guided by a teacher is being shown, in effect, a good solution to imitate, which is a far easier learning problem than discovering one unaided. Researchers have shown distilled models with well under a billion parameters matching or beating models hundreds of times larger on specific tasks, by learning from the larger model's outputs and reasoning traces rather than from raw labels. The student does not need the teacher's capacity to find the behaviour, only enough capacity to reproduce it, and reproducing a known good behaviour takes far less than discovering it.

This is also why distillation pairs so naturally with the current era of reasoning models. A large reasoning model that thinks through problems in long chains is expensive, but its reasoning traces can be used to teach a small model to reason similarly. Some of the most capable small models available were produced exactly this way, by distilling the reasoning behaviour of a frontier model into a compact one. It is a major reason that strong reasoning is no longer confined to the largest, costliest models.

Distillation versus quantization and pruning

Distillation is one of three main ways to make a model smaller and cheaper, and confusing them is common, so it helps to see the distinction clearly. Quantization shrinks a model by storing its existing weights in fewer bits, lowering precision without changing the model's structure. Pruning shrinks a model by removing weights or connections judged unimportant, cutting the existing model down. Both of these operate on the same model, compressing what is already there.

Distillation is different in kind: it does not compress the existing model at all. It trains a brand-new, smaller model to imitate the behaviour of the large one. If quantization is saving a large file in a more compact format and pruning is deleting the parts you do not need, distillation is having the expert train an apprentice from scratch. Because they work on different principles, they are complementary rather than competing: a common production recipe distills a large teacher into a smaller student, then quantizes and prunes that student further, stacking the techniques to reach maximum efficiency. Distillation builds a new efficient model; the others trim an existing one, and together they get you further than any alone.

The variants, and a competitive tension

The basic recipe has several forms. The most common is offline distillation, where a finished teacher is frozen and the student trains on its saved outputs, which is simple and stable. In online distillation, teacher and student train together, allowing more adaptation at the cost of trickier tuning. In self-distillation, a model teaches itself, with deeper layers guiding shallower ones or later checkpoints guiding earlier training. And in multi-teacher distillation, a student learns from several teachers at once. For language models specifically, the teacher's knowledge can be transferred through its next-token distributions, through the text and reasoning traces it generates, or through its internal representations, and mixing hard and soft supervision often works better than either alone.

There is a competitive and legal tension worth naming, because it has become a live issue. If you can access a powerful closed model through an API, you can, in principle, use its outputs to train your own student model, distilling a proprietary system you were never given the weights to. This is powerful and also contentious: model providers generally prohibit using their outputs to train competing models, and there have been public accusations of labs distilling one another's models. Distillation, which began as an innocuous compression trick, has become entangled with questions of intellectual property and competitive advantage, precisely because it is such an effective way to copy a model's behaviour without copying the model.

The short version

Knowledge distillation trains a small student model to mimic a large teacher model, producing a compact model that keeps much of the teacher's capability at a fraction of the cost. Its key mechanism is dark knowledge: instead of training on hard labels that give only the correct answer, the student trains on the teacher's soft probability distributions, which encode how the possibilities relate to one another, and this richer signal lets the student generalise better than if trained on raw data. A temperature setting softens the teacher's output to expose more of that relational structure. Because imitating a known good behaviour is far easier than discovering it, a distilled small model can outperform much larger ones on specific tasks, which is how strong reasoning has been compressed into cheap models. Distillation differs from quantization and pruning, which shrink an existing model, because it trains a new one, and the three are often combined.

The idea to hold onto is that distillation works because a teacher's uncertainty is not noise but a compressed lesson about the structure of a task, so a student that learns from the teacher's full probability distribution inherits some of its judgment, not just its answers, which is how frontier capability gets packed into models small enough to actually run. The big models set the frontier; distillation is how that frontier reaches your laptop, your phone, and the cheap API call, by having the giants teach compact prodigies to imitate them.

Common questions

What is knowledge distillation? Knowledge distillation is a technique for making AI models smaller and cheaper by training a compact student model to imitate the behaviour of a large, capable teacher model. Rather than training the small model on raw data alone, you train it to reproduce the teacher's outputs, so it captures much of the teacher's capability at a fraction of the size and cost. It was introduced for model compression and is now standard practice for producing deployable models, delivering large cost reductions with only a small loss in quality, which is a major reason capable small models exist.

What is dark knowledge in distillation? Dark knowledge is the extra information contained in a teacher's soft predictions, its full probability distribution over the possibilities, that a plain correct answer does not carry. When a teacher labels an image 85 percent cat, 10 percent dog, and 5 percent car, it reveals how it sees the options relate: a cat is closer to a dog than to a vehicle. Hard labels (100 percent cat) discard this relational structure. Training the student on the soft distribution transfers the teacher's whole way of organising the problem, not just the answer, which is why distilled students generalise better than the same small model trained on raw labels.

Why does distillation use soft labels instead of hard labels? Because soft labels carry far more information. A hard label states only the correct class and treats all wrong answers as equally wrong. A teacher's soft label is a probability distribution that shows how the classes relate, which ones are similar, which distinctions are subtle, and how confident to be. This relational structure, the dark knowledge, teaches the student a shadow of the teacher's understanding rather than only its conclusion. As a result, a student trained on soft labels learns the shape of the task, not just the answers, and generalises better than one trained on hard labels alone.

What is the temperature in knowledge distillation? Temperature in distillation is a setting that softens the teacher's output distribution to expose more of its dark knowledge. A well-trained teacher is often very confident, putting almost all probability on one answer and hiding the small but meaningful probabilities on related options. Dividing the teacher's raw scores by a temperature before converting them to probabilities flattens the distribution, making those subtle relationships visible so the student can learn from them. It is the same mechanism that controls randomness in text generation, used here for a different purpose: a higher temperature transmits more of the teacher's nuanced understanding to the student.

How is distillation different from quantization and pruning? All three shrink models, but they work differently. Quantization stores an existing model's weights in fewer bits, lowering precision without changing structure. Pruning removes unimportant weights or connections from an existing model. Both compress the model that is already there. Distillation instead trains a brand-new, smaller model to imitate the large one, so it builds a fresh model rather than trimming an existing one. Because they operate on different principles, they are complementary: a common recipe distills a teacher into a student, then quantizes and prunes that student further, combining all three for maximum efficiency.

Can a distilled small model beat a larger model? On specific tasks, yes. A small model trained directly on data is limited by its own capacity to discover good solutions, but a small model guided by a teacher is shown a good solution to imitate, which is a much easier learning problem. Researchers have demonstrated distilled models with under a billion parameters matching or beating models hundreds of times larger on particular tasks, by learning from the teacher's outputs and reasoning traces. The student does not need the teacher's capacity to find the behaviour, only enough to reproduce it, and reproducing a known behaviour takes far less capacity than discovering it.

Is it legal to distill a model you access through an API? It is contested. Technically, you can use a model's API outputs to train your own student model, distilling a proprietary system without ever having its weights. But model providers generally prohibit using their outputs to train competing models in their terms of service, and there have been public accusations of labs distilling one another's models. So while distillation from an API is technically feasible and widely discussed, doing it with a commercial model typically violates the provider's terms and raises intellectual-property questions. Distillation's effectiveness at copying behaviour is exactly what makes it legally and competitively sensitive.