Bayesian Neural Networks
Weights as distributions, not fixed numbers
A standard neural network learns a single fixed value for each weight via gradient descent — one specific point in parameter space that minimizes training loss. A Bayesian neural network (BNN) instead treats every weight as a random variable with a probability distribution, placing a prior distribution over weights before training and, via Bayes' rule, computing a posterior distribution over weights after observing the training data: $P(w \mid D) \propto P(D \mid w) \, P(w)$. Predictions are then made by integrating (marginalizing) over this entire posterior distribution of plausible weight settings, rather than using just one.
Why exact Bayesian inference is intractable here
For a network with millions or billions of parameters, computing the true posterior $P(w|D)$ exactly is intractable — the normalizing constant (the 'evidence' $P(D)$) requires integrating over the entire high-dimensional weight space, which has no closed form for anything but the simplest models. This intractability is the central practical challenge of Bayesian deep learning, and essentially the entire field is a search for good-enough approximations to a posterior that can never be computed exactly at any meaningful scale.
Approximate Bayesian inference methods for neural networks
| Method | Core idea | Trade-off |
|---|---|---|
| Variational inference (Bayes by Backprop) | Fit a simple distribution (e.g. Gaussian) to approximate the true posterior | Computationally efficient; approximation can be crude |
| Monte Carlo Dropout | Reinterpret dropout as approximate variational inference | Very cheap, minimal code change; weaker theoretical grounding |
| Deep ensembles | Train multiple networks, treat the ensemble as a posterior approximation | Simple and effective; not a formal Bayesian posterior |
| MCMC / Hamiltonian Monte Carlo | Sample directly from the true posterior via simulated dynamics | More faithful to true Bayesian inference; very expensive at scale |
| Laplace approximation | Fit a Gaussian around the trained (MAP) weights using local curvature | Cheap post-hoc addition to an already-trained network |
Variational inference: the practical workhorse
Variational inference reframes the intractable exact posterior computation as a tractable optimization problem: choose a family of simple, parameterized distributions $q_\phi(w)$ (commonly independent Gaussians per weight) and find the member of that family closest to the true posterior, measured by KL divergence. Minimizing this divergence is equivalent to maximizing the Evidence Lower Bound (ELBO), which can be optimized with ordinary gradient-based methods using the reparameterization trick to allow gradients to flow through the sampling step. 'Bayes by Backprop' (Blundell et al., 2015) is the canonical practical implementation of this idea for neural networks.
import torch
import torch.nn as nn
class BayesianLinear(nn.Module):
"""A linear layer where each weight has a learned Gaussian distribution."""
def __init__(self, in_features, out_features):
super().__init__()
self.weight_mu = nn.Parameter(torch.randn(out_features, in_features) * 0.1)
self.weight_logvar = nn.Parameter(torch.full((out_features, in_features), -5.0))
def forward(self, x):
std = torch.exp(0.5 * self.weight_logvar)
eps = torch.randn_like(std)
weight_sample = self.weight_mu + eps * std # reparameterization trick
return x @ weight_sample.T
def kl_divergence_to_standard_normal(mu, logvar):
# KL(q(w) || N(0, 1)), the regularizing term in the ELBO
return -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())What a BNN buys you beyond a point estimate
Why BNNs haven't replaced standard deep learning at scale
Despite their theoretical appeal, BNNs remain far less common in production deep learning than their conceptual elegance might suggest. The approximations needed to make inference tractable (mean-field variational families, assuming independence between weights) often underestimate true posterior uncertainty, and the doubled parameter count plus the need for multiple forward-pass samples at inference time imposes real compute and latency costs. For most practical uncertainty needs, cheaper approximations — deep ensembles, MC Dropout, or conformal prediction's distribution-free guarantees — deliver much of the practical benefit without the full implementation and training complexity of a genuinely Bayesian treatment.
What's Next?
Bayesian neural networks apply the Bayesian framework to deep learning's typical parametric function form; a complementary nonparametric approach to the same underlying goal — a principled distribution over functions with built-in uncertainty — is offered directly by Gaussian processes.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.