Snapshot & Neural Network Ensembles

Ensembling without N training runs

Ensembling — averaging predictions from multiple models — reliably improves accuracy and robustness over any single model, because independently-trained models make different errors that partially cancel out when combined. The standard way to build an ensemble of neural networks is to train N separate networks (different initializations, maybe different architectures) and average their predictions at inference time. This works, but it costs N times the training compute.

Snapshot ensembles get much of the ensembling benefit from a single training run, by saving multiple checkpoints ("snapshots") of the same model at different points during training and ensembling those checkpoints together — instead of training N independent models, you train one model N times "cheaper" and harvest diversity from its trajectory through weight space.

Cyclic learning rates make this possible

The trick that makes snapshots diverse rather than redundant is a cyclic learning rate schedule. Training starts with a relatively high learning rate, then is annealed down (often via cosine decay) over a fixed cycle length, allowing the model to settle into a local minimum. At the end of the cycle, a snapshot of the model's weights is saved. Then — instead of continuing to decay — the learning rate is reset to a high value again, which kicks the model out of that minimum and sends it exploring a different region of the loss landscape, where it settles into a different minimum over the next cycle.

Repeating this several times across a single training run produces several snapshots, each sitting in a distinct minimum with distinct error patterns, despite all coming from the same training run and the same final architecture. Because the learning rate restarts to a high value rather than continuing to decrease, these are often called warm restarts (as in SGDR — Stochastic Gradient Descent with Warm Restarts).

Ensembling the snapshots

At inference time, each saved snapshot runs a forward pass on the input, and their output probabilities (or logits) are averaged, exactly as with any ensemble of independently trained models. Since the snapshots came from different minima, they tend to disagree on the examples that are genuinely ambiguous or near the model's decision boundary, which is exactly where ensembling helps most — their agreements reinforce confidence, their disagreements average out individual errors.

The cost is a fraction of training N separate models: one training run, slightly extended to accommodate several cycles, versus N full independent training runs. Inference cost is the same as any ensemble of that size — N forward passes at prediction time — so the savings are specifically in training compute, not inference compute.

Snapshot ensembles vs. independently trained ensembles

Where this fits among ensembling strategies

Snapshot ensembling sits alongside other cheap-ensembling tricks like checkpoint averaging (simply averaging the weights of several late-training checkpoints, rather than their output predictions) and stochastic weight averaging (SWA), which averages weights over a cyclic or constant learning rate phase to find a wider, flatter minimum that tends to generalize better than any single snapshot. All of these trade some of the ensembling benefit of fully independent models for a large reduction in the compute needed to get most of the benefit.

What's next

Snapshot ensembles combine checkpoints of a model trained on raw features. Many real-world systems need to combine learned knowledge with structured, explicit facts instead of just numeric predictions. The next lesson, Knowledge Graphs in AI, covers how entities, relations, and graph structure are used to ground and complement learned models.