Mixed Precision Training

Precision is a resource, not a given

Neural network training traditionally used 32-bit floating point (FP32) for every number involved — weights, activations, gradients. FP32 offers wide range and fine precision, but it's expensive: it uses twice the memory of a 16-bit format, and modern GPUs can often execute 16-bit operations substantially faster than 32-bit ones using dedicated hardware (such as NVIDIA's Tensor Cores).

Mixed precision training uses lower-precision 16-bit formats for the bulk of computation — matrix multiplications in the forward and backward pass — while keeping select values in FP32 where precision actually matters, like the master copy of the weights and certain accumulations. Done correctly, this gives a substantial speedup and memory reduction with little to no loss in final model quality.

FP16 vs. BF16: the two 16-bit formats

Both formats use 16 bits total, but split them differently between exponent (range) and mantissa (precision).

FP16 (half precision) uses 5 exponent bits and 10 mantissa bits. It has fine precision within its range but a *narrow* dynamic range — values much smaller than about 6×10⁻⁵ or larger than about 65,504 overflow or underflow to zero or infinity. Gradients in deep networks routinely fall outside this range, which is why FP16 training requires a workaround.

BF16 (bfloat16) uses 8 exponent bits (the same as FP32) and only 7 mantissa bits. It has the *same* dynamic range as FP32 — so overflow and underflow are far less of a concern — at the cost of less precision for any given value. In practice, BF16 is simpler to use because it rarely needs the extra safeguards FP16 requires, which is why it has become the default choice on hardware that supports it (TPUs, and NVIDIA GPUs from the Ampere generation onward).

Loss scaling: FP16's workaround for underflow

Because FP16 gradients can underflow to zero before they're even used, loss scaling multiplies the loss by a large constant (e.g. 1024 or higher) before the backward pass. By linearity of differentiation, every gradient in the backward pass gets scaled by that same factor, pushing small gradient values up into FP16's representable range. Before the optimizer applies the update, the gradients are divided back down by the same factor to undo the scaling.

Dynamic loss scaling automates this: it starts with a large scale factor, and if it ever detects an overflow (a gradient becoming inf or NaN), it skips that update and halves the scale factor; if many consecutive steps pass without overflow, it periodically increases the scale factor again. This removes the need to hand-tune a fixed scale value. BF16, with its wider range, generally doesn't need loss scaling at all.

What stays in FP32, and why

Even in mixed precision training, certain values are deliberately kept in full FP32 precision: the master copy of the model weights is typically maintained in FP32 and updated by the optimizer, with a 16-bit copy cast out for the forward/backward computation — this avoids the accumulated rounding error of repeatedly applying tiny updates to a low-precision weight. Batch normalization statistics and certain reduction operations (like softmax and layer norm) are also commonly kept in FP32, since summing many values is exactly where low-precision rounding error compounds most.

FP32 vs. FP16 vs. BF16

What's next

Mixed precision changes how numbers are represented during training, but another factor determines whether training even gets off the ground: how the weights are set before training starts. The next lesson, Weight Initialization Strategies, covers why naive initialization schemes break deep networks before a single gradient step happens, and how Xavier/Glorot and He initialization fix it.