Knowledge Distillation
Compressing a large model's behavior into a small one
Knowledge distillation (Hinton et al., 2015) trains a smaller 'student' model to mimic a larger, more capable 'teacher' model, with the goal of capturing much of the teacher's performance at a fraction of the inference cost, latency, and memory footprint. The key insight is that a model's final hard predictions (just the top class or token) throw away a lot of information the teacher has learned — the full probability distribution over all possible outputs encodes relative confidences and similarities that are themselves a richer training signal than a single correct label.
Soft targets and temperature
Instead of training the student purely on ground-truth hard labels, distillation trains it to match the teacher's full output probability distribution — the soft targets. A temperature parameter $T > 1$ is typically applied to both the teacher's and student's logits before softmax, which flattens the distribution and reveals more of the relative probabilities assigned to non-top classes (how similar the teacher thought '7' and '1' were, even if it predicted '7'). The student is trained with a combined loss: $L = \alpha \cdot L_{hard}(\text{student}, \text{true label}) + (1-\alpha) \cdot T^2 \cdot L_{soft}(\text{student}, \text{teacher})$, balancing learning from ground truth and learning from the teacher's richer distribution.
import numpy as np
def softmax(logits, T=1.0):
z = logits / T
z = z - z.max()
e = np.exp(z)
return e / e.sum()
def distillation_loss(student_logits, teacher_logits, true_label_idx, T=2.0, alpha=0.5):
student_soft = softmax(student_logits, T=T)
teacher_soft = softmax(teacher_logits, T=T)
# KL divergence between teacher and student soft distributions
soft_loss = np.sum(teacher_soft * (np.log(teacher_soft + 1e-9) - np.log(student_soft + 1e-9)))
# Standard cross-entropy against the true hard label
student_hard = softmax(student_logits, T=1.0)
hard_loss = -np.log(student_hard[true_label_idx] + 1e-9)
return alpha * hard_loss + (1 - alpha) * (T ** 2) * soft_loss
teacher_logits = np.array([2.0, 1.0, 0.1, -1.0])
student_logits = np.array([1.5, 0.8, 0.3, -0.5])
loss = distillation_loss(student_logits, teacher_logits, true_label_idx=0)
print(f"distillation loss: {loss:.4f}")Dark knowledge in the non-top predictions
Distillation vs other compression techniques
| Technique | What it reduces | Requires a teacher model? |
|---|---|---|
| Knowledge distillation | Parameter count via a smaller architecture | Yes |
| Quantization | Precision of weights/activations (e.g. fp16 to int8) | No |
| Pruning | Number of weights within the same architecture | No |
| Low-rank factorization | Effective rank of weight matrices | No |
Distillation in modern LLM pipelines
Beyond classic classification settings, distillation is now widely used to train smaller, cheaper LLMs by generating training data from a large frontier model's outputs and fine-tuning a smaller model on them — sometimes called response-based distillation, since it transfers behavior through generated text examples rather than matching logit distributions directly (which requires access to the teacher's internals, often unavailable for API-only models). This is a major reason smaller open-weight models have closed so much of the capability gap with frontier models on common tasks — much of their training data is itself generated or curated using a larger model's outputs.
A distilled model inherits the teacher's blind spots and biases
What's Next?
Distillation is really a specific instance of a broader teacher-student training setup — the same supervisory relationship shows up, with different mechanics, well beyond simple model compression.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.