System Balance: Matching ML Workloads to Hardware

The fastest GPU in the world won't save a starved pipeline

It's tempting to treat hardware upgrades as the default fix for a slow ML system: training too slow, buy a bigger GPU; inference too slow, buy more of them. This works only when compute is actually the bottleneck. In a large fraction of real systems, the GPU spends much of its time idle, waiting on data to arrive from disk, waiting on a slow preprocessing step on the CPU, or waiting on memory bandwidth to shuttle weights in and out of fast on-chip memory. Buying a faster GPU in that situation buys you a faster idle GPU — the wall-clock time barely moves.

Three places a system can bottleneck

Think of an ML workload as data flowing through three resources: compute (the arithmetic units doing matrix multiplies), memory bandwidth (how fast data moves between memory and compute), and I/O (how fast data arrives from disk, network, or a data loader). A workload is compute-bound when the arithmetic itself is the limiting factor — large matrix multiplies on well-batched data tend to land here. It's memory-bound when the model does relatively little arithmetic per byte of data moved — this is common in autoregressive LLM decoding, where each step only computes one new token but has to move the entire set of cached weights and activations through memory. It's I/O-bound when the bottleneck is getting data to the chip at all — common in training pipelines reading from slow storage, or in systems with unoptimized data loaders.

GPU utilization numbers can lie

A common diagnostic mistake is trusting a 'GPU utilization: 95%' metric as proof the GPU is the bottleneck. That metric typically measures whether any kernel is running on the GPU at any given moment, not whether the GPU is doing useful, well-parallelized work. A GPU can show high utilization while running tiny, poorly batched kernels that leave most of its compute units idle — technically 'busy,' practically wasteful. Profiling actual throughput (tokens/sec, images/sec, samples/sec against the theoretical peak for that hardware) is a more honest signal than the utilization percentage alone.

Batching as the lever that shifts the bottleneck

Batch size is one of the most effective levers for moving a workload from memory-bound to compute-bound. Processing one sample at a time means the GPU loads weights into fast memory, does a small amount of work with them, and discards them — mostly paying the memory cost without amortizing it. Processing many samples per batch reuses those same loaded weights across more computation, raising arithmetic intensity and pushing utilization toward the compute roof. This is exactly why inference serving systems batch requests together even at the cost of a few milliseconds of added latency per request — the throughput gain from better hardware utilization usually outweighs the latency cost, up to a point where batches get so large that the latency cost itself becomes the problem.

Diagnosing the bottleneck

What's next

Understanding compute, memory, and I/O bottlenecks requires a mental model of what the hardware is actually doing with each operation in a neural network. The next lesson, Neural Computation Fundamentals, breaks down how neural computation maps to real hardware operations like matrix multiplies and FLOPs.