Model Parallelism Strategies

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

When one device isn't enough

Training or even just loading a large model often exceeds what a single accelerator can hold or compute in reasonable time. A 70-billion-parameter model in 16-bit precision needs roughly 140GB just for weights, before gradients, optimizer states, and activations — far beyond any single GPU's memory. Model parallelism is the umbrella term for strategies that split a single model's computation and/or parameters across multiple devices, as distinct from data parallelism, which keeps a full model copy on every device and splits the training *data* instead.

The main axes of model parallelism

There isn't one way to split a model — different strategies cut along different dimensions of the computation, and production training runs typically combine several simultaneously. Pipeline parallelism splits the model by layer, assigning different consecutive layers to different devices, so data flows through devices like a factory line. Tensor parallelism splits individual weight matrices within a layer across devices, so a single matrix multiplication is computed cooperatively by multiple GPUs. Data parallelism (technically orthogonal to model parallelism, but always combined with it in practice) replicates the whole model and splits batches of training examples across devices, synchronizing gradients afterward.

Parallelism strategies compared

Strategy	What's split	Communication pattern	Main benefit
Data parallelism	Training batch across replicas	All-reduce gradients after each step	Scales training throughput with more devices
Pipeline parallelism	Layers across devices (sequential stages)	Activations passed stage to stage	Fits models too deep for one device's memory
Tensor parallelism	Individual weight matrices within a layer	Frequent, low-latency all-reduce within a layer	Fits individual layers too wide for one device
ZeRO / FSDP (sharded data parallelism)	Optimizer states, gradients, and optionally parameters across replicas	Gather/scatter shards as needed per step	Removes most of data parallelism's memory redundancy

Why they're combined, not chosen exclusively

Frontier model training typically uses a 3D (or higher) parallelism strategy, nesting multiple techniques: tensor parallelism within a node (since it requires very fast, low-latency interconnects like NVLink), pipeline parallelism across nodes within a cluster (tolerating higher latency between pipeline stages), and data parallelism across many such pipeline+tensor-parallel groups to scale total throughput. Choosing the right combination and degree of each is one of the most consequential infrastructure decisions in large-scale training, directly affecting both achievable model size and training speed (measured in tokens processed per second per dollar of compute).

parallelism_layout.txt

text

Example: training a large model across 64 GPUs

  Tensor-parallel group (4 GPUs, within one node, NVLink):
    GPU0 GPU1 GPU2 GPU3  <- each holds 1/4 of each weight matrix

  Pipeline-parallel stages (4 stages, each stage = one TP group):
    [Stage0: layers 1-8]  ->  [Stage1: layers 9-16]  ->  [Stage2: layers 17-24]  ->  [Stage3: layers 25-32]
     (GPU0-3)                  (GPU4-7)                   (GPU8-11)                  (GPU12-15)

  Data-parallel replicas (4 copies of the full pipeline, 16 GPUs each):
    Replica A (GPU0-15)  Replica B (GPU16-31)  Replica C (GPU32-47)  Replica D (GPU48-63)
    -> gradients all-reduced across replicas after each step

The guiding principle: match communication cost to interconnect speed

The reason these strategies are nested in a specific order (tensor parallel innermost, data parallel outermost) is communication frequency and volume. Tensor parallelism requires an all-reduce on every single layer's forward and backward pass — it needs the fastest possible interconnect, so it's confined to GPUs within the same physical node. Data-parallel gradient synchronization happens once per training step and tolerates much higher latency, so it can span across nodes or even across clusters with slower networking.

More parallelism isn't free

Every parallelism strategy introduces communication overhead and potential device idle time (pipeline 'bubbles' while later stages wait for earlier ones to produce activations). Poorly tuned parallelism configurations can leave expensive accelerators idle a significant fraction of the time, which is why 'model FLOPs utilization' — the fraction of theoretical peak compute actually used — is a closely tracked metric in large training runs, often the difference between a well-engineered and a poorly-engineered training job at the same cluster size.

What's next?

Pipeline parallelism and tensor parallelism each deserve a closer look on their own — the first for how it manages the sequential dependency between layers without leaving devices idle, and the second for how it splits individual matrix operations while keeping results numerically correct.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →