Bayesian Inference & MAP Estimation

Bayes' theorem is a simple rule for inverting conditional probabilities: $P(H|D) = \frac{P(D|H) \cdot P(H)}{P(D)}$

Each term has a name and a job:

- $P(H)$ — the prior: your belief about hypothesis $H$ before seeing any data
- $P(D|H)$ — the likelihood: how probable the observed data $D$ is, assuming $H$ is true
- $P(D)$ — the evidence (or marginal likelihood): the overall probability of the data, averaged over all hypotheses — it acts as a normalizing constant
- $P(H|D)$ — the posterior: your updated belief about $H$ after seeing $D$

In words: posterior is proportional to likelihood times prior. The data reshapes your prior belief in proportion to how well each hypothesis explains what you observed.

Maximum Likelihood Estimation picks the parameter value that makes the observed data most probable, ignoring any prior belief entirely: $\hat{\theta}_{MLE} = \arg\max_\theta P(D|\theta)$

This is the workhorse of classical statistics and much of machine learning — fitting a logistic regression by minimizing cross-entropy loss is mathematically equivalent to MLE under a Bernoulli likelihood. MLE is simple and consistent (it converges to the true parameter with enough data), but with limited data it can overfit badly, since it has no mechanism to pull estimates toward sensible values.

MAP estimation fixes MLE's blind spot by folding in a prior: $\hat{\theta}_{MAP} = \arg\max_\theta P(D|\theta) \cdot P(\theta) = \arg\max_\theta [\log P(D|\theta) + \log P(\theta)]$

MAP finds the single most probable parameter value given both the data and prior knowledge. The prior acts as a regularizer: a Gaussian prior on model weights centered at zero is mathematically identical to L2 regularization (ridge regression), and a Laplace prior is identical to L1 regularization (lasso). This is a deep and useful fact — regularization isn't an arbitrary trick, it's MAP estimation with a specific choice of prior.

As you collect more data, the likelihood term dominates and MAP converges toward the MLE solution — the prior matters most when data is scarce, which is exactly when you need help the most.

The Naive Bayes classifier is a direct, practical application of Bayes' theorem. To classify an email as spam or not, it computes $P(\text{spam}|\text{words})$ using Bayes' theorem, with the "naive" simplifying assumption that all word features are conditionally independent given the class. This assumption is almost never literally true, but it makes the math tractable — multiply individual word likelihoods together instead of modeling their full joint distribution — and in practice the classifier still performs remarkably well, especially on text classification.

The class with the highest posterior probability wins — which makes Naive Bayes classification itself a MAP estimation problem, just applied to a discrete class label rather than a continuous parameter.