Gradient Clipping

When one bad batch wrecks the whole model

Occasionally during training, a single mini-batch produces a gradient whose magnitude is enormous — orders larger than typical. Applied directly, that gradient's update can throw the model's weights into a region of the loss landscape it can never recover from: weights become NaN, loss spikes and never comes back down, or training destabilizes entirely. This is the exploding gradient problem, and it's especially common in recurrent neural networks, where gradients are repeatedly multiplied across many time steps and can grow exponentially, and in deep transformers early in training before the loss landscape has smoothed out.

Gradient clipping is a simple, cheap safeguard: before the optimizer applies an update, check whether the gradient's magnitude exceeds a threshold, and if so, scale it back down. It's not a fix for why gradients explode — it's a circuit breaker that prevents any single bad step from destroying the model.

Norm clipping: rescale the whole gradient vector together

Gradient norm clipping computes the overall norm of the entire gradient vector (across all parameters, treated as one long vector), and if that norm exceeds a threshold max_norm, rescales every component proportionally so the new norm equals exactly max_norm. Because every component is scaled by the same factor, the gradient's direction is unchanged — only its length is capped.

This is the standard, most widely used form of clipping in practice. It's the default in PyTorch's clip_grad_norm_ and is standard in training recipes for RNNs, LSTMs, and transformers alike.

Value clipping: a cruder, per-element alternative

Gradient value clipping instead clamps each individual gradient element independently to a fixed range, such as [−1, 1], regardless of the other elements. Unlike norm clipping, this *does* change the gradient's direction, because different components can get clipped by different relative amounts — it's a blunter tool. It's simpler to reason about per-element but less commonly recommended for modern deep learning, since distorting the gradient's direction can itself introduce a different kind of instability. It still appears in some older RNN training recipes and in settings where extreme outlier values in a few dimensions specifically (rather than overall magnitude) is the concern.

Norm clipping vs. value clipping

How to choose a clipping threshold

There's no universal correct value — it depends on model architecture, batch size, and learning rate. A practical approach: log the gradient norm at every step for the first few hundred steps of training *without* clipping, look at the typical range, and set max_norm somewhat above the typical value but below the occasional spikes you want to suppress. Many training recipes simply reuse a value that's worked for similar architectures (1.0 is extremely common for transformers) and verify empirically that training stays stable.

What's next

Gradient clipping protects against numerical instability during training, but a separate class of speedups comes from changing the numerical precision of training itself — trading some precision for a large gain in speed and memory. The next lesson, Mixed Precision Training, covers how FP16 and BF16 formats accomplish this without destroying model accuracy.