Text-to-Image Diffusion Pipelines

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

From noise to image: the diffusion pipeline, end to end

A text-to-image diffusion system is a pipeline of several distinct trained components working together, not one monolithic model. Understanding the full pipeline matters because each stage can be swapped, fine-tuned, or debugged independently — a common source of confusion is treating 'the diffusion model' as a single black box when it's really text encoder, denoiser, and image decoder chained together.

Stage 1: text encoding

The input prompt is tokenized and passed through a text encoder — commonly a CLIP text encoder, or in newer pipelines a full language model (e.g. T5) — producing a sequence of embedding vectors that represent the prompt's meaning. These embeddings aren't used to generate pixels directly; they're injected into the denoising network as a conditioning signal via cross-attention, letting every image-generation step 'look at' the relevant words in the prompt.

Stage 2: iterative denoising in latent space

Most production pipelines (Stable Diffusion and descendants) don't generate pixels directly — they operate in a compressed latent space produced by a separate autoencoder, which shrinks a 512x512 image down to something like a 64x64x4 latent grid, cutting compute dramatically. Starting from pure Gaussian noise in this latent space, a U-Net (or, in newer models, a diffusion transformer) repeatedly predicts the noise component present at the current step and subtracts a fraction of it, guided by the text embeddings through cross-attention layers. Over tens of steps (or as few as 1-4 in distilled models), the latent progressively resolves from noise into a coherent, prompt-aligned representation.

ddim_sampling_loop.py

python

import torch

@torch.no_grad()
def sample(unet, text_embeds, latent_shape, num_steps, guidance_scale, scheduler):
    latents = torch.randn(latent_shape)  # start from pure noise

    for t in scheduler.timesteps[:num_steps]:
        # Classifier-free guidance: run once conditioned, once unconditioned
        noise_cond = unet(latents, t, context=text_embeds)
        noise_uncond = unet(latents, t, context=None)
        noise_pred = noise_uncond + guidance_scale * (noise_cond - noise_uncond)

        latents = scheduler.step(noise_pred, t, latents)  # denoise one step

    return latents  # decode with VAE decoder to get the final image

Classifier-free guidance: turning up prompt adherence

The sampling loop runs the denoiser twice at each step — once conditioned on the text prompt, once with no conditioning — and extrapolates away from the unconditioned prediction toward the conditioned one. The guidance_scale controls how far: a scale of 1 ignores the extrapolation (weaker prompt adherence, more natural-looking but less literal images), while higher scales (7-15 is typical) push the output to match the prompt more aggressively, at some cost to image diversity and naturalism if pushed too far.

Stage 3: decoding back to pixels

The final denoised latent is passed through the decoder half of the variational autoencoder (VAE) used in stage 2, upsampling the compact latent grid back into a full-resolution RGB image. This decoder is trained separately, typically with a combination of reconstruction and perceptual/adversarial losses, and is reused unchanged across many different diffusion checkpoints — it's responsible for fine texture quality, not for following the prompt.

Pixel-space vs latent-space diffusion

Property	Pixel-space diffusion	Latent-space diffusion (Stable Diffusion)
Denoising resolution	Full image resolution (e.g. 512x512x3)	Compressed latent (e.g. 64x64x4)
Compute cost per step	Very high	Much lower — the main efficiency win
Extra component needed	None	Pretrained VAE encoder/decoder
Example models	early DDPM, Imagen (partially)	Stable Diffusion, SDXL, Flux

VAE artifacts are a separate failure mode from denoiser errors

Because decoding happens in a separate, lossy compression step, certain generation artifacts — garbled small text, slightly malformed fine detail, color-banding in gradients — originate in the VAE's reconstruction fidelity, not in the denoising U-Net's prompt understanding. Swapping in a higher-fidelity VAE decoder can fix these artifacts without touching the diffusion model at all, which is a useful debugging distinction when diagnosing output quality issues.

What's Next?

This same text-conditioned latent-diffusion pipeline is the substrate that techniques like ControlNet build on top of, adding extra spatial conditioning signals beyond text to give much finer control over the generated layout and structure.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →