When you train a classifier with cross-entropy loss, you are, quite literally, measuring how "surprised" the model is by the correct answer. Information theory — originally developed by Claude Shannon to quantify communication over noisy channels — gives us the precise mathematical tools for this: entropy, cross-entropy, and KL divergence. Nearly every classification loss function in deep learning is one of these three ideas in disguise.

Entropy: the average surprise of a distribution

Entropy measures how unpredictable a distribution is, in bits (when using $\log_2$): $H(P) = -\sum_x P(x) \log_2 P(x)$.

A fair coin ($P(\text{heads})=P(\text{tails})=0.5$) has maximum uncertainty: $H = -(0.5\log_2 0.5 + 0.5\log_2 0.5) = -(0.5(-1) + 0.5(-1)) = 1$ bit. A biased coin with $P(\text{heads})=0.9, P(\text{tails})=0.1$ is much more predictable: $H = -(0.9\log_2 0.9 + 0.1\log_2 0.1) \approx -(0.9(-0.152) + 0.1(-3.32)) \approx 0.137 + 0.332 = 0.469$ bits — less than half the uncertainty of the fair coin.

python

Cross-entropy: measuring surprise using the wrong distribution

Cross-entropy measures the average surprise when you use one distribution $Q$ (the model's predictions) to describe outcomes actually generated by another distribution $P$ (the true labels): $H(P, Q) = -\sum_x P(x) \log_2 Q(x)$.

Suppose the true label is "cat" with certainty ($P = [1, 0]$ for classes [cat, dog]) and the model predicts $Q = [0.8, 0.2]$. Cross-entropy is $-(1 \cdot \log_2 0.8 + 0 \cdot \log_2 0.2) = -\log_2 0.8 \approx 0.322$ bits. If the model instead predicted $Q = [0.4, 0.6]$ — leaning the wrong way — cross-entropy jumps to $-\log_2 0.4 \approx 1.32$ bits. The worse the model's prediction, the higher the cross-entropy, which is exactly why it works as a training loss: minimizing it pushes $Q$ toward $P$.

python

Note

PyTorch's and TensorFlow's cross-entropy loss functions use the natural log ($\ln$) instead of $\log_2$, measuring in "nats" instead of bits, but the mechanism is identical: penalize the model proportional to $-\log(\text{predicted probability of the true class})$. A confident correct prediction (probability near 1) gives a loss near 0; a confident wrong prediction gives a loss that explodes toward infinity.

KL divergence: the gap between two distributions

Kullback-Leibler (KL) divergence measures how much one distribution $Q$ diverges from a reference distribution $P$: $D_{KL}(P \| Q) = \sum_x P(x)\log_2 \frac{P(x)}{Q(x)} = H(P,Q) - H(P)$.

It is exactly the gap between cross-entropy and entropy — the "extra" bits of surprise caused specifically by using $Q$ instead of the true $P$. Using the cat/dog example: $H(P,Q) \approx 0.322$ bits (with $Q=[0.8,0.2]$) and $H(P) = 0$ bits (since $P=[1,0]$ has zero uncertainty), so $D_{KL}(P\|Q) = 0.322 - 0 = 0.322$ bits. When $P$ is a one-hot label, cross-entropy and KL divergence are numerically identical — which is why many classification papers use the terms almost interchangeably.

python

Note

$D_{KL}(P\|Q) \neq D_{KL}(Q\|P)$ in general — it matters which distribution is the "true" reference and which is the approximation. This asymmetry is why variational autoencoders and diffusion models are careful about which direction of KL divergence they minimize, since each direction penalizes different kinds of mismatch (e.g. covering all of $P$'s mass vs. avoiding mass where $P$ has none).

Entropy, cross-entropy, and KL divergence compared

Quantity	Formula	Interpretation	Common ML use
Entropy $H(P)$	$-\sum P(x)\log P(x)$	Inherent uncertainty of the true distribution	Decision tree splits, policy entropy bonus in RL
Cross-entropy $H(P,Q)$	$-\sum P(x)\log Q(x)$	Average surprise using the model's predictions	Classification loss, language model loss
KL divergence $D_{KL}(P\\|Q)$	$H(P,Q) - H(P)$	Extra bits lost by using $Q$ instead of $P$	VAEs, knowledge distillation, distribution matching

What's next

You now know why so many loss functions are, mathematically, variations on the same idea: minimize the surprise your model experiences. The next lesson — Optimization Algorithms: SGD, Adam & RMSprop Compared — covers how models actually search for the parameters that make that surprise as small as possible.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →