Cross-Entropy Loss Deep Dive

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Where cross-entropy comes from: information theory

Cross-entropy has a precise meaning before it's ever used as a loss function: given a true distribution $p$ and a model's predicted distribution $q$ over the same events, cross-entropy $H(p,q) = -\sum_x p(x) \log q(x)$ measures the average number of bits (using $\log_2$) or nats (using $\ln$) needed to encode events drawn from $p$ when using a code optimized for $q$ instead of $p$. If $q$ perfectly matches $p$, cross-entropy equals the entropy of $p$ itself — the theoretical minimum. Any mismatch between $q$ and $p$ adds extra bits, quantified exactly by the KL divergence: $H(p,q) = H(p) + D_{KL}(p \| q)$. For a classification target, $p$ is the one-hot true distribution (entropy zero), so minimizing cross-entropy is exactly equivalent to minimizing $D_{KL}(p\|q)$ — pushing the model's predicted distribution as close as possible to the true one.

Binary vs. categorical cross-entropy

For binary classification with true label $y \in \{0,1\}$ and predicted probability $\hat{y}$: $L = -[y \log \hat{y} + (1-y)\log(1-\hat{y})]$. For multi-class classification with $K$ classes, one-hot target $y$, and predicted distribution $\hat{y}$ (typically from a softmax): $L = -\sum_{k=1}^{K} y_k \log \hat{y}_k$. Because $y$ is one-hot, this sum collapses to a single term: $L = -\log \hat{y}_{\text{true class}}$ — the loss only looks at the probability assigned to the correct class and ignores how probability is distributed among the wrong ones (a property label smoothing deliberately changes).

cross_entropy_from_scratch.py

python

import numpy as np

def softmax(z):
    z = z - np.max(z, axis=-1, keepdims=True)  # numerical stability
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def cross_entropy_loss(logits, true_class_idx):
    probs = softmax(logits)
    # Only the true class's log-probability matters for one-hot targets
    return -np.log(probs[np.arange(len(logits)), true_class_idx] + 1e-12)

logits = np.array([[2.0, 0.5, -1.0], [0.1, 0.1, 3.0]])
targets = np.array([0, 2])
print(cross_entropy_loss(logits, targets))
# Low loss when logits strongly favor the true class, high loss otherwise

Why the gradient with softmax is so clean

The pairing of softmax activation with cross-entropy loss isn't arbitrary — it produces an unusually simple gradient. If $z$ are the pre-softmax logits and $y$ is the one-hot target, the gradient of cross-entropy loss with respect to the logits is simply $\frac{\partial L}{\partial z} = \hat{y} - y$ — predicted probabilities minus true labels, element-wise. This clean form is a direct consequence of softmax's own derivative canceling neatly against the $1/\hat{y}$ term from the log in cross-entropy's derivative. It's also why softmax+cross-entropy is implemented as a single fused operation in most frameworks (e.g. nn.CrossEntropyLoss in PyTorch expects raw logits, not post-softmax probabilities) — both for this gradient simplicity and for numerical stability.

A classic numerical bug: double softmax

Applying softmax yourself and then feeding the result into a loss function that *also* applies softmax internally (like PyTorch's CrossEntropyLoss, which expects raw logits) silently produces a wrong, overly-flattened loss. The symptom is usually slow or stalled training with no obvious error — always check whether your loss function expects logits or probabilities before writing the forward pass.

The log-sum-exp trick for numerical stability

Computing softmax naively — exponentiating raw logits directly — overflows for even moderately large logit values, since $e^{50}$ already exceeds standard float32 range. The standard fix, used internally by every major framework, is to subtract the maximum logit before exponentiating: $\text{softmax}(z)_i = \frac{e^{z_i - \max(z)}}{\sum_j e^{z_j - \max(z)}}$. This is mathematically identical to the unshifted version (the max cancels in the ratio) but keeps every exponentiated value in a safe numerical range. Combined cross-entropy+softmax implementations go further and compute the loss directly from logits via the log-sum-exp identity, avoiding an intermediate division entirely.

Loss variant	Target type	Typical pairing
Binary cross-entropy	Single scalar probability	Sigmoid output, binary/multi-label classification
Categorical cross-entropy	One-hot vector	Softmax output, single-label multi-class
Sparse categorical cross-entropy	Integer class index	Same as categorical, just a memory-saving encoding
KL divergence loss	Full probability distribution	Distillation, soft targets, label smoothing

Cross-entropy and maximum likelihood are the same thing

Minimizing cross-entropy loss over a dataset is mathematically identical to maximizing the likelihood of the data under the model (maximum likelihood estimation), assuming the model's output is interpreted as a categorical probability distribution. This is why cross-entropy shows up as the default loss across such a wide range of probabilistic models — it isn't a heuristic choice, it's the direct consequence of treating classification as density estimation over discrete outcomes.

What's Next?

Cross-entropy assumes the model's predicted probabilities are meaningful as probabilities — whether that assumption actually holds, and how to check or fix it, is the subject of calibration and temperature scaling.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →