Federated Learning

Training without ever collecting the data

Distributed training, as usually practiced, assumes the data is already centralized — sitting in cloud storage, accessible to every GPU in the training cluster. Federated learning starts from the opposite assumption: the data lives on millions of individual devices (phones, hospital servers, individual organizations) and should never leave those devices as raw data, either for privacy reasons, regulatory reasons, or because moving that much raw data centrally simply isn't practical. Federated learning trains a shared model by moving the computation to the data instead of moving the data to the computation.

The basic federated round

A federated learning round works roughly like this: a central server sends the current model to a sample of participating devices. Each device trains the model locally for a few steps using only its own local data, producing an updated set of weights (or a weight update/gradient). Devices send back only that update — never the raw data — to the server. The server aggregates the updates from all participating devices (commonly by averaging them) to produce a new global model, which is sent out again in the next round. This repeats for many rounds until the global model converges.

Why communication, not compute, is the bottleneck

In data-center distributed training, the interconnect between GPUs is fast and reliable. In federated learning, the 'interconnect' is each participant's often-slow, often-unreliable internet connection, and devices may be on metered mobile data or battery power, making frequent large uploads expensive or simply unwelcome. This flips the usual optimization priority: federated learning systems optimize hard for communication efficiency — sending model updates as few times and as compactly as possible — rather than for raw compute throughput. Common techniques include compressing updates before sending them, having each device run several local training steps per round instead of one (trading a bit of staleness for fewer communication rounds), and sampling only a subset of available devices each round rather than waiting on every device to respond.

Non-IID data: the statistical headache unique to federated learning

In centralized training, you control how data is split across workers, so each worker's local data roughly resembles the whole dataset's distribution. In federated learning, each device's data reflects that one user's or one organization's behavior — a phone's keyboard data reflects one person's vocabulary, a hospital's patient data reflects its local demographics. This 'non-IID' (not independently and identically distributed) data makes the naive averaging of updates less reliable than in standard distributed training, because each device is effectively training on a biased local slice, and devices can end up pulling the shared model in different directions. Federated learning algorithms (FedAvg and its many variants) spend considerable effort specifically compensating for this.

Centralized distributed training vs. federated learning

What's next

Federated learning makes reproducing a training run even harder than usual, since the data itself is never in one place to inspect. The next lesson, ML Reproducibility Engineering, covers the broader discipline of making ML experiments and training runs reliably repeatable in the first place.