Knowledge Distillation
What is it?
Knowledge distillation trains a small 'student' model to imitate a large 'teacher' model, capturing much of its ability at a fraction of the size.
Explain like I'm 5
Why was it created?
Large models are accurate but expensive to run. Distillation was created to compress that capability into smaller, faster models suitable for cheaper or on-device deployment.
Where is it used?
- Shrinking models for production
- On-device and mobile AI
- Speeding up inference
- Creating smaller open models
Why should developers care?
Many fast, affordable models you use are distilled versions of bigger ones, so it's key to why capable AI can run cheaply.
How does it work?
The teacher runs on training inputs and produces outputs (often soft probabilities, not just labels). The student is trained to match those outputs, learning the teacher's nuanced behavior rather than just hard answers.
Real-world example
A company distills its large chat model into a smaller one that answers most queries at a fraction of the cost and latency.
Common use cases
- Cost and latency reduction
- Edge/mobile deployment
- Building smaller specialist models
- Compressing an ensemble into one model
Advantages
- Much smaller, faster, cheaper student
- Keeps most of the teacher's quality
- Enables on-device AI
- Can compress ensembles
Disadvantages
- Student rarely matches the teacher fully
- Needs a strong teacher and data
- Extra training pipeline to maintain
- Can inherit teacher's biases
When should you use it?
When you need a cheaper, faster model and can tolerate a small quality drop from a larger one.
When should you avoid it?
When you need maximum quality and can afford the large model, or when no strong teacher exists.
Alternatives
Related terms
Interview questions
Beginner
- What are the teacher and student in distillation?
- Why distill a model at all?
Intermediate
- Why train on soft probabilities instead of just labels?
- How does distillation compare to quantization?
Senior
- What is the role of temperature in the distillation loss?
- How would you distill an ensemble into a single model?
Common misconceptions
- "The student equals the teacher" — it usually trades some quality for size.
- "Distillation and quantization are the same" — distillation trains a new smaller model; quantization shrinks an existing one's precision.
Fun facts
- Using the teacher's soft probabilities ('dark knowledge') carries more information than hard labels.
- Many popular compact models are distilled from much larger ones.
Timeline
- 2015 — Hinton et al. formalize knowledge distillation
- 2019 — DistilBERT shows big compression with little quality loss
Learning resources
Quick summary
Knowledge distillation trains a small student model to mimic a large teacher, keeping most of its quality at much lower cost.
Cheat sheet
- Student mimics teacher
- Learns from soft probabilities
- Smaller, faster, cheaper
- Slight quality trade-off