Training & Adaptationintermediate12 min · 8 steps
Knowledge Distillation
How a large teacher model trains a small student: soft labels, temperature, on-policy and self-distillation, and why API operators now treat distillation as a threat.
datatokensignalresponserequesterror
Key ideaStep 1 of 8
Small models learn from big ones
Many small models aren't trained on raw web data alone. They learn from a larger teacher: its outputs, its logits, or its reasoning traces. That's a big reason today's small open models punch above their size.
The documented cases are open models. Closed labs don't publish small-model recipes, so "Haiku is distilled from Opus" is plausible, not documented.
| Student | Teacher signal |
|---|---|
| Gemma 2 (2B, 9B), all Gemma 3 | Distilled from a larger teacher in pretraining |
| Llama 3.2 1B / 3B | Logits from Llama 3.1 8B and 70B |
| DeepSeek-R1 distills | ~800K samples generated by R1 |
| NVIDIA competitive-coding model | 477,642 traces from GLM-5.2 (MIT) |
What changed
- +Teacher
- +Student
- +Knowledge flowing between them
Lesson 5 of 8 in Training & Adaptation