Knowledge distillation, in the context of artificial intelligence, is a neural model compression technique that transfers the knowledge learned by a large model, called the teacher model, to a smaller model, called the student model. The goal is to obtain a system that is lighter and faster to run while keeping performance as close as possible to the original model. Introduced systematically in the early 2000s and later widespread with the growth of deep neural networks, this technique is now central to developing applications that must run on devices with limited resources.
The underlying principle is simple: instead of training the student model only on labeled data, you also train it to imitate the teacher model’s responses. These responses contain richer information than the original labels because they reflect the probabilities the teacher assigns to each class or each token. In this way, the student learns not only the correct answer but also how the expert model distributes its confidence across alternatives.
The two phases of the process
The process has two main phases. In the first, you train or retrieve a large teacher model, typically very accurate but expensive in terms of memory and computation. In the second, you train a more compact student model, using as the learning signal a combination of the true labels and the teacher’s predictions.
A key element is temperature, a parameter that controls the softness of the probability distributions produced by the teacher. Increasing the temperature spreads the probabilities more evenly and reveals similarities between classes that would otherwise remain hidden. The student is thus guided to capture structural relationships in the data that the labels alone do not make explicit.
Why probabilities matter more than labels
When a model assigns, for example, a high probability to a correct class and a moderate probability to a similar class, it implicitly communicates that those two categories are close. This type of information, called dark knowledge, is what makes distillation more effective than plain supervised training.
What can be transferred from teacher to student
There are several variants of the technique, which differ in what is transferred from the teacher to the student. The choice depends on compression goals, data availability, and the architectures of the models involved.
Imitating the final outputs
This is the most common form: the student directly imitates the teacher’s outputs, that is, the final probabilities. It is simple to implement and does not require access to the expert model’s internal layers.
Reproducing internal representations
In this case, the student tries to reproduce the teacher’s internal representations, that is, the activations of intermediate layers. This approach can be more effective, but it requires the architectures to be somewhat compatible.
Learning the relationships between examples
Here you do not imitate individual outputs but rather the relationships between different examples, such as the similarities computed by the teacher. The student therefore learns a global structure of the data space, not just pointwise answers.
Comparing variants by what they transfer
| Approach | What is transferred | Complexity | Main advantage |
|---|---|---|---|
| Responses | Final probabilities | Low | Simplicity of implementation |
| Features | Internal activations | Medium | Better knowledge transfer |
| Relations | Similarities between examples | High | Learning of global structures |
| Self-distillation | Knowledge of the same model | Medium | No external teacher needed |
| Multi-teacher | Outputs of multiple models | High | Greater robustness of the student |
Where it is used: from devices to language models
Knowledge distillation is widely used to bring large models to smartphones, embedded devices, and browsers, where memory and computing power are limited. It is also used in recommendation systems, speech recognition, machine translation, and compact language models.
Another advantage concerns inference times: a smaller student model responds faster and consumes less energy. This aspect is crucial in real-time applications and in services that must handle many requests simultaneously.
Reduced versions for specific tasks
In the field of large language models, distillation makes it possible to create reduced versions that maintain good performance on specific tasks, such as text classification or guided generation. These versions are often easier to deploy and control.
From model choice to data preparation
- Define the compression goal and the memory and latency constraints of the final system.
- Select or train a teacher model with high performance on the chosen task.
- Choose the student model architecture, smaller but suited to the task.
- Prepare the training dataset, with or without original labels.
- Compute the teacher’s outputs on the available examples, adjusting the temperature.
- Train the student by combining the loss toward the true labels and the loss toward the teacher.
- Evaluate the student’s performance on an independent test set.
- Optimize hyperparameters and, if necessary, repeat the cycle with a different teacher.
Frequently asked questions
What is the difference between distillation and pruning?
Pruning removes parameters or connections from an existing model, while distillation trains a new, smaller model to imitate the large one. The two techniques can be combined for further reductions.
Does distillation always reduce accuracy?
Generally, the student is less accurate than the teacher, but in some cases it can come very close or even surpass it on specific data, thanks to the regularizing effect of soft probabilities.
Do you need the original data labels?
Not strictly: there are forms of distillation that use only the teacher’s outputs. However, combining true labels and teacher predictions tends to give better results.
Does distillation work with different architectures?
Yes, the student can have a different architecture from the teacher. Response distillation is the most flexible in this respect.
What are the main drawbacks?
It requires time and resources to train the teacher and can be complex to calibrate. Moreover, the transferred knowledge depends on the quality and coverage of the data used.
Is distillation used in computer vision models?
Yes, it is very common in convolutional networks for image recognition, where it allows you to obtain lightweight models suitable for mobile devices without losing too much precision.
From glossary to projectDo language models cite your site as a source?Large language model (LLM) optimization is about getting your brand cited in Google AI Overview answers and in the main LLMs.Discover LLM optimization
Related entries:
Supervised learning Deep learning Machine learning Artificial neural network Generative artificial intelligence