ML Hardware Acceleration: GPUs, TPUs & Accelerators

It's all matrix multiplication

Strip away the architecture diagrams and nearly all of deep learning reduces to one operation performed an enormous number of times: multiplying matrices (and the closely related operation of convolving tensors, which can itself be expressed as matrix multiplication). A linear layer is a matrix multiply. Attention is a handful of matrix multiplies plus a softmax. Training multiplies that same workload by the number of parameters, the number of tokens, and the number of steps.

This single fact explains the entire hardware landscape. Chip designers aren't trying to build a general-purpose fast computer — they're trying to build the fastest possible machine for doing enormous numbers of multiply-accumulate operations in parallel, as cheaply and with as little wasted energy as possible. Every major hardware decision in ML — from GPU core counts to TPU systolic arrays to the existence of specialized accelerators — traces back to optimizing that one operation.

GPUs: parallel, general-purpose, flexible

GPUs were originally built to rasterize triangles for graphics, which turned out to require exactly the kind of massively parallel, repetitive arithmetic that neural networks also need. A modern GPU contains thousands of relatively simple cores, each capable of doing arithmetic independently, organized into groups that execute the same instruction across many data elements simultaneously (SIMD/SIMT execution).

For matrix multiplication, a GPU breaks the big matrix into tiles, assigns tiles to different cores, and runs them in parallel. Modern GPUs (NVIDIA's since Volta) also include dedicated tensor cores — hardware units purpose-built to do small matrix multiplications (e.g., 4x4 blocks) in a single operation, rather than looping through scalar multiply-adds. This is why mixed-precision training (using lower-precision formats like FP16 or BF16) is so much faster: tensor cores are specifically optimized for those formats.

GPUs remain the most flexible accelerator — they run arbitrary CUDA/compute kernels, support a vast software ecosystem, and work well for research where the computation graph changes frequently.

TPUs: a systolic array built for one job

Google's Tensor Processing Units take a more specialized approach. Instead of thousands of independent cores coordinating via shared memory, a TPU core centers on a systolic array: a large grid of simple multiply-accumulate units wired directly to their neighbors, through which data flows in a synchronized rhythm (the name comes from the pulsing, heartbeat-like movement of data, like blood through a systolic cycle).

In a systolic array, weight values are loaded into the grid once and held stationary. Activations stream in from one edge, and partial sums flow through the grid, picking up a multiply-accumulate at every cell, until finished results stream out the other side. Because data moves directly from one compute cell to its neighbor instead of constantly reading and writing to memory, the array achieves very high arithmetic density with comparatively low power draw and chip area, at the cost of flexibility — a TPU is good at exactly this choreographed data flow and worse at irregular, branching computation.

The practical consequence: TPUs tend to win on large, regular matrix-multiply-heavy workloads at scale (TPU pods interconnect many chips with very high-bandwidth links purpose-built for distributed training), while GPUs retain the edge on flexibility, software maturity, and workloads with more irregular control flow.

GPU vs. TPU: conceptual comparison

"Accelerator" means more than GPU or TPU

GPUs and TPUs get the attention, but the broader category of ML accelerators is wider. AWS builds its own chips (Trainium for training, Inferentia for inference). Several startups (Cerebras, Groq, SambaNova, Graphcore) have built chips with fundamentally different memory architectures — for example putting far more memory directly on the compute die (SRAM) rather than relying on external HBM, trading total capacity for enormously higher bandwidth. Apple's Neural Engine and similar mobile NPUs (neural processing units) accelerate inference on-device at a fraction of the power a GPU would draw, because mobile workloads prioritize energy efficiency over raw throughput.

What unifies all of these under "accelerator": each one makes a deliberate trade-off between flexibility, memory bandwidth, on-chip memory capacity, and power, tuned for a specific slice of the ML workload — training at scale, high-throughput inference, or low-power edge inference. There is no single best accelerator; there's a best accelerator for a given workload shape and deployment constraint.

What's next

Choosing the right accelerator only matters if you can reliably measure whether it's actually faster or better for your workload. The next lesson, ML Benchmarking Methodology, covers how to measure ML systems rigorously — fixed hardware, fixed data, and the pitfalls that make so many published benchmark comparisons misleading.