Autoencoders & Variational Autoencoders

Compression as a learning signal

An autoencoder is a neural network trained to do something that sounds pointless at first: reproduce its own input as its output. The trick is in the architecture. The network is forced to squeeze the input through a narrow bottleneck — a layer with far fewer dimensions than the input — before expanding back out to reconstruct it. To pass information through that bottleneck and still reconstruct accurately, the network has no choice but to learn a compressed, information-dense representation of the data's underlying structure. That compressed representation is the actual point. Reconstruction is just the training signal that forces it to exist.

Encoder, latent space, decoder

An autoencoder has three parts:

Encoder — a network that maps the high-dimensional input $x$ down to a low-dimensional vector $z$, called the latent representation or code.

Latent space — the bottleneck itself, the compressed coordinate system the encoder maps into. Its dimensionality is a hyperparameter you choose — smaller forces more aggressive compression and more abstraction.

Decoder — a network, often a mirror image of the encoder, that maps $z$ back up to a reconstruction $\hat{x}$ in the original input space.

Training minimizes the reconstruction loss — typically mean squared error for continuous data like images, or cross-entropy for binary/categorical data — between $x$ and $\hat{x}$. No labels are needed; the input is its own target, which makes this an unsupervised learning method.

What autoencoders are good for

Dimensionality reduction — the encoder alone is a learned, nonlinear alternative to PCA, useful as a preprocessing step before clustering or visualization.

Denoising — train the encoder on corrupted inputs but reconstruct the clean original; the network learns to ignore noise and reconstruct essential structure (a *denoising autoencoder*).

Anomaly detection — train only on normal data. At inference time, anomalous inputs reconstruct poorly (high reconstruction error) because the network never learned to represent them, making reconstruction error itself a usable anomaly score.

Pretraining — the compressed representation can be a useful feature extractor for downstream supervised tasks, especially when labeled data is scarce.

The variational autoencoder's probabilistic twist

A Variational Autoencoder (VAE) changes one thing: instead of the encoder outputting a single fixed point $z$, it outputs the parameters of a probability distribution — typically the mean $\mu$ and standard deviation $\sigma$ of a Gaussian. During training, $z$ is *sampled* from that distribution ($z = \mu + \sigma \cdot \epsilon$, where $\epsilon$ is random noise — the "reparameterization trick" that keeps this sampling step differentiable). The decoder then reconstructs from the sampled $z$.

The loss function gets a second term alongside reconstruction loss: a KL divergence penalty that pulls every encoded distribution toward a standard normal distribution ($\mu = 0, \sigma = 1$). This regularizer forces the entire latent space to be smooth, continuous, and densely packed around the origin — with no gaps and no isolated clusters. Now, sampling a random point from a standard normal and feeding it to the decoder reliably produces a plausible, novel output. The VAE has turned a compression tool into a generative model.

Autoencoder vs. VAE vs. GAN vs. diffusion model

Try it yourself — see how data flows through encoder and decoder layers

What's next

To see the adversarial alternative to the VAE's probabilistic approach, go to gans. For the iterative-denoising approach that currently leads in generation quality, see diffusion-models.