Teacher-Student Training
A broader framing than compression alone
Teacher-student training is the general pattern underlying knowledge distillation, but it extends well beyond 'shrink a model.' In a teacher-student setup, one model (or an ensemble, or even a non-neural oracle) supplies supervisory signal — soft labels, pseudo-labels, rankings, or corrections — that trains a second model. The teacher and student don't need to differ in size at all; they can be the same architecture, different architectures entirely, or even operate on different input modalities.
Self-training and pseudo-labeling
One major variant is self-training: a model is first trained on a small labeled dataset, then used as a 'teacher' to generate pseudo-labels for a much larger pool of unlabeled data, and a 'student' (often the same architecture, sometimes larger) is trained on the combination of real labels and pseudo-labels. This is effectively teacher-student training where the teacher and student start from the same lineage, and it's a core technique for leveraging large amounts of unlabeled data when labeling is expensive — common in speech recognition, machine translation for low-resource languages, and semi-supervised computer vision.
Confirmation bias in iterative self-training
Noisy Student and ensemble teachers
The Noisy Student method (Xie et al., 2020) made a specific, influential refinement: train a teacher on labeled data, use it to generate pseudo-labels on a large unlabeled set, then train a student — equal or larger in size — on the combined data with heavy data augmentation and regularization (dropout, stochastic depth) applied only to the student, not the teacher. This asymmetry forces the student to generalize robustly from the teacher's labels rather than overfitting to them, and iterating this process (student becomes next round's teacher) produced then-state-of-the-art ImageNet results using large amounts of unlabeled data.
def self_training_round(teacher, unlabeled_data, confidence_threshold=0.9):
"""Generate high-confidence pseudo-labels from the current teacher."""
pseudo_labeled = []
for x in unlabeled_data:
probs = teacher.predict_proba(x)
confidence = probs.max()
if confidence >= confidence_threshold:
pseudo_labeled.append((x, probs.argmax()))
return pseudo_labeled
def train_student(student, labeled_data, pseudo_labeled_data, augment_fn):
combined = labeled_data + pseudo_labeled_data
for x, y in combined:
x_aug = augment_fn(x) # augmentation applied, teacher never saw it
student.train_step(x_aug, y)
return student
# Iterating: student becomes the teacher for the next round, on a larger
# or fresh unlabeled pool, is the core loop behind Noisy Student-style training.Teacher-student variants
| Variant | Teacher source | Primary goal |
|---|---|---|
| Knowledge distillation | Larger pretrained model | Compress to a smaller, faster model |
| Self-training / pseudo-labeling | Model trained on limited labels | Leverage unlabeled data |
| Noisy Student | Same lineage, asymmetric augmentation | Robust generalization from unlabeled data |
| Ensemble distillation | Multiple models combined | Transfer ensemble accuracy into one model |
RLHF's reward model is a teacher too
What's Next?
Teacher-student setups are one way to get useful training signal without expensive labels. A related but distinct family of techniques gets useful representations with no labels or teacher at all, by having a model learn structure directly from the data's own internal relationships.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.