Probability Theory for Machine Learning

ML is built on uncertainty, not certainty

A classifier doesn't output "this is a cat" — it outputs "87% probability this is a cat." A language model doesn't pick the next word deterministically — it samples from a probability distribution over the vocabulary. Probability theory is the formal language for reasoning about exactly this kind of uncertainty, and it underlies loss functions, Bayesian methods, generative models, and evaluation metrics alike.

Random variables: numbers that depend on chance

A random variable $X$ is a variable whose value is determined by the outcome of some random process. Rolling a die is a random variable with possible values $\{1,2,3,4,5,6\}$, each with probability $1/6$ (a discrete random variable). A model's predicted probability for a class, or a pixel's noise value in a diffusion model, can take any value in a continuous range (a continuous random variable).

Each random variable has a distribution — a complete description of how likely each value (or range of values) is. For a discrete variable, this is a probability mass function (PMF); for continuous, a probability density function (PDF).

Expectation: the long-run average

The expectation (or expected value) of a random variable is its probability-weighted average: $E[X] = \sum_x x \cdot P(X=x)$ for discrete variables. For a fair six-sided die, $E[X] = \frac{1}{6}(1+2+3+4+5+6) = 3.5$ — notably not even a possible outcome, but the long-run average over many rolls.

For the biased coin above, encode heads as 1 and tails as 0: $E[X] = 1(0.7) + 0(0.3) = 0.7$. This is exactly why a loss function like cross-entropy is defined as an expectation — it's the average penalty over the whole data distribution, not any single example.

Variance: how spread out the outcomes are

Variance measures how far values typically land from the expectation: $\text{Var}(X) = E[(X - E[X])^2]$. For the fair die ($E[X]=3.5$): compute $(1-3.5)^2=6.25$, $(2-3.5)^2=2.25$, $(3-3.5)^2=0.25$, $(4-3.5)^2=0.25$, $(5-3.5)^2=2.25$, $(6-3.5)^2=6.25$. Average them: $(6.25+2.25+0.25+0.25+2.25+6.25)/6 = 17.5/6 \approx 2.92$.

In ML, variance shows up directly in the bias-variance tradeoff (how much a model's predictions swing across different training sets) and in batch normalization, which explicitly normalizes activations by their variance.

Joint and conditional probability: how variables relate

Joint probability $P(X, Y)$ describes the chance of two events happening together — e.g. $P(\text{rain}, \text{traffic jam})$. Conditional probability $P(X \mid Y)$ describes the chance of $X$ given that $Y$ has already happened: $P(X \mid Y) = \frac{P(X,Y)}{P(Y)}$.

Suppose $P(\text{rain}) = 0.2$, $P(\text{traffic jam}) = 0.3$, and $P(\text{rain}, \text{traffic jam}) = 0.12$. Then $P(\text{traffic jam} \mid \text{rain}) = 0.12 / 0.2 = 0.6$ — a 60% chance of a jam given rain, much higher than the unconditional 30%. This is the backbone of Bayes' theorem, Naive Bayes classifiers, and how a language model computes $P(\text{next token} \mid \text{previous tokens})$.

Core probability concepts and their ML role

What's next

Probability gives you the language for uncertainty; the next lesson — Information Theory: Entropy, Cross-Entropy & KL Divergence — shows how that uncertainty is actually measured and turned directly into the loss functions that train almost every modern classifier and language model.