Stable Diffusion Internals
Three trained models, not one
Stable Diffusion is often described as a single model, but it's actually three separately trained neural networks wired together: a VAE (variational autoencoder) that compresses images to and from a latent space, a text encoder (CLIP) that turns prompts into conditioning embeddings, and a U-Net that performs the actual iterative denoising. Each has a distinct job and can, in principle, be swapped or fine-tuned independently of the others — which is exactly what techniques like LoRA fine-tuning and VAE replacement exploit in practice.
The VAE: why diffusion happens in latent space, not pixel space
A 512x512 RGB image has 786,432 numbers. Running dozens of U-Net denoising steps directly over that many pixels, at every training step, across millions of images, is prohibitively expensive. Stable Diffusion's VAE encoder compresses the image by a factor of 8 in each spatial dimension and down to 4 channels, producing a 64x64x4 latent — roughly 48x fewer values. Diffusion happens entirely in this compressed space; the VAE decoder only runs once, at the very end, to turn the final denoised latent back into pixels. This is the single biggest reason Stable Diffusion is tractable on consumer GPUs while earlier pixel-space diffusion models were not.
The forward (noising) process the model is trained to reverse
Training doesn't need generated images at all — it needs real images with synthetic noise added. The forward process defines, for any real latent $z_0$ and timestep $t$, a closed-form way to produce a noised version:
$$z_t = \sqrt{\bar\alpha_t}\, z_0 + \sqrt{1-\bar\alpha_t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$
where $\bar\alpha_t$ is a scheduled constant that decreases from near 1 (almost no noise) at $t=0$ to near 0 (almost pure noise) at the final timestep. The U-Net is trained to predict exactly the noise $\epsilon$ that was added, given the noised latent $z_t$, the timestep $t$, and the text conditioning. Minimizing the mean-squared error between predicted and true noise is the entire training objective — simple, stable, and parallelizable across random timesteps and random training images.
import torch
def training_step(unet, vae_encoder, text_encoder, image, prompt, alpha_bar_schedule):
with torch.no_grad():
z0 = vae_encoder(image).sample() # compress image to latent
text_embeds = text_encoder(prompt)
t = torch.randint(0, len(alpha_bar_schedule), (1,))
alpha_bar_t = alpha_bar_schedule[t]
noise = torch.randn_like(z0)
z_t = (alpha_bar_t.sqrt() * z0) + ((1 - alpha_bar_t).sqrt() * noise) # forward process
predicted_noise = unet(z_t, t, context=text_embeds)
loss = torch.mean((predicted_noise - noise) ** 2) # simple MSE objective
return lossCross-attention is where the prompt actually enters the U-Net
Components and what fine-tuning each one changes
| Component | Job | Effect of fine-tuning it |
|---|---|---|
| VAE | Pixel <-> latent compression | Changes fine texture/detail fidelity, not content |
| Text encoder (CLIP) | Prompt -> conditioning embedding | Changes how prompt wording is interpreted |
| U-Net | Iterative denoising, prompt-conditioned | Changes subject matter, style, composition (LoRA targets this) |
The scheduler is a separate design choice from the trained model
What's Next?
Understanding this three-part structure — compress, condition, denoise — is the foundation for every extension built on top of it: ControlNet injects extra structural conditioning into the U-Net, LoRA fine-tunes the U-Net's attention weights cheaply, and newer diffusion transformer architectures replace the U-Net's convolutional backbone while keeping the same overall latent-diffusion recipe intact.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.