Distributed Training: Data & Model Parallelism

Two different problems that both get called 'scaling training'

There are two distinct reasons a single GPU stops being enough for training. The first: the dataset is large and training would simply take too long on one device, even though the model itself fits comfortably in memory. The second: the model itself is too large to fit in a single GPU's memory at all, regardless of how patient you are. These are different problems with different solutions, and conflating them leads teams to reach for the wrong tool — adding more GPUs the wrong way doesn't fix a model that doesn't fit on one of them.

Data parallelism: same model, different data, on every GPU

Data parallelism solves the first problem. Each GPU holds a full copy of the model and processes a different slice of the training batch simultaneously. After each GPU computes gradients on its slice, the gradients are averaged (synchronized) across all GPUs before the weight update is applied, so every copy of the model stays identical. This is conceptually simple and scales well as long as the model fits on one device — doubling the number of GPUs roughly doubles throughput, modulo the cost of synchronizing gradients across devices, which grows as you add more GPUs and eventually becomes its own bottleneck.

Model (and pipeline) parallelism: splitting the model itself

When a model's parameters alone don't fit on one GPU's memory, data parallelism doesn't help — every GPU would still need a full copy that doesn't fit. Model parallelism instead splits the model across devices. Tensor parallelism splits individual large matrix multiplies across GPUs, so each device computes a portion of the same layer's output. Pipeline parallelism instead assigns different layers (or groups of layers) to different GPUs, so data flows through GPU 0's layers, then GPU 1's layers, and so on, like an assembly line. Pipeline parallelism's main efficiency challenge is keeping every GPU busy — naively, GPU 1 sits idle until GPU 0 finishes its portion of the first batch, creating a 'bubble' of wasted time that pipelining schedules (processing multiple micro-batches in flight at once) are specifically designed to minimize.

Communication overhead is the shared enemy of every strategy

Every parallelism strategy introduces communication between devices — synchronizing gradients in data parallelism, passing activations between pipeline stages, exchanging partial results in tensor parallelism — and that communication competes for the same network bandwidth connecting the GPUs. This is why large training clusters invest heavily in high-bandwidth, low-latency interconnects between GPUs (far faster than a typical data-center network), and why the way you partition work across GPUs is chosen partly based on which communication pattern that interconnect handles best — tensor parallelism, with its frequent small exchanges, generally needs the fastest, closest interconnect, while data parallelism's periodic gradient averaging tolerates somewhat looser connectivity.

Data parallelism vs. model/pipeline parallelism

What's next

Distributed training assumes all the data lives somewhere centrally accessible to the training cluster. That assumption breaks down when data is sensitive and scattered across millions of user devices — the exact scenario the next lesson, Federated Learning, is built to handle.