Model Compression: Quantization, Pruning & Distillation

Why a trained model is often the wrong size to ship

A model optimized purely for accuracy during training tends to be larger, slower, and more memory-hungry than production constraints allow. A 70-billion-parameter language model might be state-of-the-art on benchmarks but cost too much per request to serve profitably, or be physically unable to run on a phone, a browser, or an embedded device. Model compression is the set of techniques for reducing a model's size and compute cost while keeping as much of its accuracy as possible.

The three dominant techniques — quantization, pruning, and knowledge distillation — attack the problem from different angles, and production systems frequently combine more than one.

Quantization: fewer bits per number

Neural network weights and activations are typically stored as 32-bit floating point numbers (FP32) during training. Quantization reduces this precision — to FP16, INT8, or even 4-bit integers — cutting memory footprint and often speeding up inference, since lower-precision arithmetic is cheaper on most hardware and more values fit in cache and memory bandwidth at once.

There are two main flavors: post-training quantization (PTQ), which converts an already-trained model's weights without retraining (fast, simple, small accuracy hit), and quantization-aware training (QAT), which simulates low-precision arithmetic during training so the model learns weights that are robust to the eventual precision loss (more effort, better accuracy retention, especially important going below INT8). The risk with aggressive quantization is that outlier weight values can get clipped or rounded badly enough to meaningfully hurt accuracy, which is why most modern quantization schemes use per-channel or per-group scaling factors rather than one scale for the whole tensor.

Pruning: removing weights that don't matter much

Pruning removes individual weights, neurons, or entire structural components (channels, attention heads, layers) that contribute little to the model's output, based on some importance criterion — most simply, the magnitude of the weight itself (small weights are assumed less important). Pruned weights are typically set to zero, and if the pruning is structured (removing whole channels or heads rather than scattered individual weights), the resulting model can actually run faster on standard hardware, not just use less memory.

Unstructured pruning (removing arbitrary individual weights) achieves higher sparsity ratios for the same accuracy loss, but the resulting sparse matrix doesn't map cleanly onto dense matrix-multiplication hardware, so you need specialized sparse-matrix kernels to realize a speedup — otherwise you've shrunk the model on disk without actually making it faster. Structured pruning (removing whole neurons, channels, or heads) sacrifices some potential sparsity but produces a smaller dense model that runs faster on ordinary GPUs and CPUs without special kernel support — which is why structured pruning is more common in practice despite being less theoretically efficient.

Knowledge distillation: teaching a small model to mimic a big one

Knowledge distillation trains a smaller 'student' model to reproduce the behavior of a larger, already-trained 'teacher' model — not just matching the teacher's final hard labels, but matching its full output distribution (the 'soft labels,' or logits before the final decision), which carries richer information about how confident and how it ranks alternative answers.

This works because the teacher's soft outputs encode more signal than ground-truth labels alone — a teacher that outputs 70% cat, 25% dog, 5% other is teaching the student something about the relationship between cat and dog that a hard 'cat' label alone doesn't convey. DistilBERT, a widely used distilled version of BERT, retains around 97% of BERT's performance on benchmark tasks while being roughly 40% smaller and 60% faster — a representative result for how much capacity can often be recovered through distillation relative to training a small model from scratch on hard labels alone.

Quantization vs. pruning vs. distillation

Choosing and combining techniques

In practice these aren't mutually exclusive. A common production pipeline distills a large teacher into a smaller student architecture, then prunes the student to remove redundant structure, then quantizes the pruned student to INT8 for final deployment — each step compounding the size and latency reduction while the earlier steps' fine-tuning helps the model recover accuracy lost at each stage. The right starting point depends on your constraint: if you're memory-bound but have fast hardware, quantization alone often gets most of the win cheaply; if you need a fundamentally smaller architecture (for a mobile or edge deployment), distillation is usually necessary because quantization and pruning alone can't shrink a model past a certain architectural floor.

What's next

A compressed model still has to run somewhere, serving real requests under real latency constraints. The next lesson, Model Serving & Inference Systems, covers the architecture of inference servers, batch vs. online inference, and how to roll out and roll back model versions safely.