Restricted Boltzmann Machines (RBMs)

The architecture: two layers, no within-layer connections

A Restricted Boltzmann Machine has two layers of binary units: a visible layer v (representing the data, e.g., pixel values) and a hidden layer h (representing learned features). Every visible unit connects to every hidden unit, but — and this is the "restriction" that gives RBMs their name and their tractability — there are no connections within the visible layer and none within the hidden layer.

That restriction is what makes RBMs practical: given the visible units, all hidden units are conditionally independent of each other (and vice versa), so you can sample an entire layer in one parallel step instead of one unit at a time.

An energy-based model with one specific structure

An RBM is a direct special case of the energy-based model from the previous lesson. It defines an energy function E(v, h) = -aᵗv - bᵗh - vᵗWh, where W is the weight matrix connecting visible and hidden units, and a, b are bias terms. Lower energy means a more probable joint configuration of visible and hidden units, exactly as in the general EBM framework — RBMs just restrict the energy function to this specific bilinear form, which is what makes the conditional independence (and therefore efficient sampling) possible.

Stacking RBMs: Deep Belief Networks and pretraining

Before 2012, training deep networks end-to-end from random initialization often failed to converge well — gradients vanished or the optimization landscape was too hard to navigate directly. Hinton's 2006 breakthrough was to train a stack of RBMs greedily, one layer at a time: train an RBM on the raw data, then treat its learned hidden-layer activations as the "data" for a second RBM, and so on. This stack, called a Deep Belief Network, gave each layer a sensible unsupervised starting point before any supervised fine-tuning happened.

This layer-wise pretraining was the technique that first convincingly demonstrated deep networks (more than two or three layers) could be trained successfully at all — directly motivating the deep learning resurgence that followed.

RBM pretraining then vs. now

What they're still good for

RBMs still appear in a few niches: as a simple collaborative-filtering model (RBMs were part of a well-known approach in the Netflix Prize competition), in some physics-inspired modeling contexts where the Boltzmann-distribution framing has direct physical meaning, and in educational settings, since an RBM is small enough to implement and train from scratch while still illustrating energy-based reasoning, latent variable models, and MCMC sampling in one package.

What's next

RBMs learn distributed, latent features through an energy function. Self-Organizing Maps take a different unsupervised route entirely — instead of a probability or energy, they directly learn a low-dimensional, topologically organized map of the data's structure.