Energy-Based Models

Scoring instead of modeling probability directly

Most generative models try to directly output or sample from a probability distribution. Energy-Based Models (EBMs) take an indirect but very general route: learn a scalar function E(x), called the energy, where low energy means "x looks like real data" and high energy means "x looks fake or unlikely." The model never has to normalize anything during training — it just has to assign lower energy to real data than to other configurations.

The energy function connects to a probability distribution through the Boltzmann distribution: p(x) = exp(-E(x)) / Z, where Z is a normalizing constant (the partition function) that sums exp(-E(x)) over every possible x.

The intractable partition function problem

The entire difficulty of EBMs is this: Z = Σ exp(-E(x)) summed (or integrated) over every possible x is almost never computable. For images, "every possible x" is an astronomically large space. Without Z you can compare relative likelihoods of two points (which has lower energy) but you can't compute an actual normalized probability, and — more importantly for training — you can't directly maximize log-likelihood, because its gradient depends on Z.

Training: contrastive methods and Langevin dynamics

The standard training trick sidesteps computing Z directly. The gradient of the log-likelihood turns out to depend only on an expectation over the model's own distribution — so instead of computing it analytically, you approximate it by drawing samples from the current model and comparing their energy against real data's energy. The update rule becomes simple and intuitive: push down energy on real data, push up energy on samples the model currently generates.

Generating those samples usually uses Markov Chain Monte Carlo (MCMC), most commonly Langevin dynamics: start from noise and take small steps that follow the energy function's gradient (going "downhill" toward lower energy), with a bit of injected noise at each step to keep exploring rather than collapsing to one point.

EBMs compared to other generative families

Where the energy-based view matters today

Few production systems are described as "an EBM" outright, but the framework's influence is everywhere. Score-based and diffusion models — currently the dominant approach to image and audio generation — are deeply connected to energy-based training (learning a score function, the gradient of a log-density, is mathematically adjacent to learning an energy function). EBMs also see direct use in structured prediction tasks (where outputs have complex internal dependencies, like sequence labeling) and in physics-inspired modeling, where an explicit energy has natural scientific meaning.

What's next

Restricted Boltzmann Machines are a historically important, tightly constrained special case of an energy-based model — simple enough to train with the specific technique of contrastive divergence, and they were the original building block for the pretraining methods that kicked off modern deep learning.