The equations behind neural networks are exact mathematical statements, but computers don't do exact arithmetic — they use floating point numbers, which represent real numbers with a finite number of bits and therefore finite precision. Most of the time this doesn't matter. But at the extreme values that show up constantly in deep learning — very large exponentials, very small probabilities, very deep chains of multiplication — that finite precision becomes the difference between a model that trains and one that silently outputs NaN.

Floating point precision has hard limits

A standard 32-bit float (float32) can represent roughly 7 decimal digits of precision and has a maximum magnitude around $3.4 \times 10^{38}$ before it overflows to infinity. Crucially, precision isn't uniform: float32 can distinguish $1.0000001$ from $1.0000002$, but it cannot distinguish $10{,}000{,}000.1$ from $10{,}000{,}000.2$ — at larger magnitudes, the gaps between representable numbers get wider.

This matters directly in ML: adding a tiny gradient update to a very large accumulated weight can simply have no effect at all, because the update falls below the representable precision at that magnitude.

python

Exploding and vanishing values

Deep networks repeatedly multiply numbers across many layers — weights during the forward pass, gradients during backpropagation (via the chain rule). If the typical multiplier is consistently greater than 1, repeated multiplication compounds exponentially and overflows to infinity (exploding). If it's consistently less than 1, it compounds down toward zero (vanishing), and gradients become too small to meaningfully update early layers.

Concretely: a weight of $1.5$ multiplied through 50 layers gives $1.5^{50} \approx 6.4 \times 10^8$ — large but survivable in float32. A weight of $2.0$ through 50 layers gives $2^{50} \approx 1.1\times10^{15}$, uncomfortably close to float32's precision limits. And a weight of $0.5$ through 50 layers gives $0.5^{50} \approx 8.9 \times 10^{-16}$ — already indistinguishable from zero at float32 precision, which is exactly how vanishing gradients silently stall training.

python

Note

Careful weight initialization schemes (Xavier/Glorot, He initialization) and normalization layers (batch norm, layer norm) exist specifically to keep activations and gradients hovering near a magnitude of 1 as they pass through dozens or hundreds of layers — directly preventing the exponential explode/vanish dynamic described above. Gradient clipping (capping gradient magnitude at a threshold) is the blunt-force backup for when explosion still happens despite these safeguards.

The log-sum-exp trick: why softmax needs it

Softmax converts raw scores (logits) into probabilities: $\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$. The problem: if any logit $z_i$ is even moderately large (say 1000), $e^{1000}$ overflows float32 instantly to infinity, and infinity divided by infinity gives NaN — a silently broken training run.

The fix is the log-sum-exp trick: subtract the maximum logit from every logit before exponentiating. Since $\frac{e^{z_i}}{\sum_j e^{z_j}} = \frac{e^{z_i - m}}{\sum_j e^{z_j - m}}$ for any constant $m$ (it cancels out algebraically), choosing $m = \max_j z_j$ guarantees every exponent is $\le 0$, so every $e^{z_i-m} \le 1$ — no overflow is possible, and the result is mathematically identical.

python

Note

Framework functions like PyTorch's log_softmax followed by nll_loss (or the fused cross_entropy) apply log-sum-exp internally and compute $\log(\text{softmax}(z_i))$ directly as $z_i - m - \log\sum_j e^{z_j-m}$, avoiding ever computing a raw probability that could underflow to exactly 0 (which would make $\log(0) = -\infty$ and break the loss). This is why you should almost always use a framework's fused log-softmax + loss function rather than computing softmax and log separately by hand.

Common numerical stability failure modes and fixes

Failure mode	Cause	Common fix
Overflow to infinity / NaN	Large exponentials (e.g. raw softmax logits), unclipped gradients	Log-sum-exp trick, gradient clipping, lower learning rate
Underflow to zero	Repeated multiplication of values < 1 across many layers	Careful weight initialization, normalization layers, log-space computation
Silent precision loss	Adding small updates to large accumulated values	Mixed-precision training with careful loss scaling, gradient accumulation
Vanishing gradients	Chain rule multiplying many small derivatives (e.g. sigmoid)	ReLU-family activations, residual connections, normalization

What's next

Numerical stability is about taming the arithmetic of a single training run; the next lesson — The Central Limit Theorem for Machine Learning — zooms out to the statistical assumptions that quietly justify why averaging over batches, samples, and runs behaves the way practitioners expect it to.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →