Pipeline Parallelism

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Splitting a model by layer, across devices

Pipeline parallelism assigns consecutive groups of layers — 'stages' — to different devices. Device 0 might hold layers 1-8, device 1 layers 9-16, and so on. During a forward pass, device 0 computes its layers and sends the resulting activations to device 1, which continues the computation, and so on down the pipeline. The backward pass runs the reverse: gradients flow from the last stage back to the first. This directly solves the problem of a model being too deep (too many layers' worth of parameters) to fit on one device, at the cost of introducing a sequential dependency between stages.

The bubble problem

The naive version of pipeline parallelism has an obvious inefficiency: while device 1 is computing on a batch's activations, device 0 has nothing to do until device 1 passes something back (in the backward pass) or until the next batch arrives. These idle periods are called pipeline bubbles, and in the simplest schedule, bubble time grows with the number of pipeline stages — more stages mean proportionally more idle time relative to useful compute, which defeats much of the purpose of adding stages in the first place.

Micro-batching: the standard fix

The practical solution is to split each training batch into several smaller micro-batches and pipe them through the stages back-to-back, so that while device 1 processes micro-batch 1, device 0 is already working on micro-batch 2. This keeps more devices busy more of the time. The GPipe schedule does this straightforwardly (all forward micro-batches, then all backward micro-batches), while the PipeDream / interleaved 1F1B (one-forward-one-backward) schedule interleaves forward and backward micro-batch processing to shrink bubble time and reduce the peak memory needed to hold activations for later backward passes.

pipeline_schedule.txt

text

GPipe-style schedule, 4 stages, 4 micro-batches (F = forward, B = backward):

Stage0: F1 F2 F3 F4                      B4 B3 B2 B1
Stage1:    F1 F2 F3 F4                B4 B3 B2 B1
Stage2:       F1 F2 F3 F4           B4 B3 B2 B1
Stage3:          F1 F2 F3 F4     B4 B3 B2 B1
                 ^-- bubble region (early stages idle waiting for later ones)

1F1B interleaved schedule overlaps forward/backward earlier, shrinking the bubble:
Stage0: F1 F2 F3 B1 F4 B2    B3    B4
Stage1:    F1 F2 B1 F3 B2 F4 B3    B4
...

Pipeline scheduling strategies

Schedule	Bubble size	Peak activation memory	Complexity
Naive (no micro-batching)	Largest — proportional to stage count	Low	Simple
GPipe (all-forward then all-backward)	Reduced, but still grows with stage count	High — holds all micro-batch activations until backward	Moderate
1F1B / interleaved	Smallest among standard schedules	Lower — activations freed sooner	More complex scheduling logic

More micro-batches trade bubble time for memory

Increasing the number of micro-batches per training step shrinks the relative bubble overhead (more useful work fills the pipeline), but each in-flight micro-batch's activations have to be kept around until its backward pass runs, so memory use grows with the number of micro-batches in flight. Choosing micro-batch count and pipeline depth together is a real tuning exercise, not a default setting.

Pipeline parallelism needs activation checkpointing to stay memory-feasible

Even with good scheduling, holding activations for multiple in-flight micro-batches across many pipeline stages can be memory-prohibitive at scale. Most production pipeline-parallel training combines it with activation checkpointing (recomputing activations during the backward pass instead of storing them) to keep memory usage within what's actually available on each device.

What's next?

Pipeline parallelism solves the 'too many layers to fit' problem; a complementary technique is needed when even a single layer's weight matrices are too large for one device to hold — that's the problem tensor parallelism is built to solve, by splitting individual matrix operations rather than whole layers.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →