Knowledge Distillation Solved a Latency Problem Fine-Tuning Couldn't

The constraint was non-negotiable: a real-time content moderation classifier needed to respond in under 50ms per request on commodity CPU hardware, no exceptions, because it sat directly in a user-facing posting flow. Our large fine-tuned classifier hit well under that at over 90% accuracy — on GPU inference. On the CPU hardware the production environment actually used, it blew past 200ms.

The direct approach that didn't work

The obvious move was training a smaller model directly on the same labeled dataset, aiming for something fast enough on CPU. We tried three smaller architectures. All hit the latency target. All landed meaningfully lower on accuracy — the smaller models, trained directly on the raw labels, just didn't have the capacity to learn the nuanced decision boundaries the large model had found. We were stuck trading accuracy for speed in a way that made the client understandably nervous about false negatives on a moderation task.

Training a small model directly on the raw labels and training a small model to mimic a large model's decisions are two different tasks, and the second one works much better.

What distillation actually bought us

Instead of training the small model on the original binary labels, we trained it to match the large model's full output probability distribution — soft labels instead of hard ones. The large model's confidence scores carried information about near-miss cases that the raw labels alone didn't, and the small model learned those decision boundaries far more effectively by mimicking them than by trying to rediscover them from scratch.

• When a smaller model trained directly on labels underperforms, try distilling from the larger model's output distribution instead of its raw predictions.
• Soft labels (full probability distributions) carry more signal than hard labels for a smaller model to learn from.
• Always benchmark latency on the actual production hardware, not GPU numbers that won't reflect the real deployment environment.
• Distillation adds a training step but no runtime cost — the deployed model is exactly as fast as any other small model of the same architecture.

The distilled small model hit 43ms on the target CPU hardware and came within two points of the large model's accuracy — a gap small enough that the client's moderation team considered it fully acceptable. Distillation isn't a universal fix for every latency problem, but it earned a permanent spot in our toolkit for exactly this shape of constraint: hard latency ceiling, accuracy you can't casually give up.