A large model training run can take days to months across thousands of accelerators. At that scale, hardware failures aren't an edge case — hitting a hardware fault, network partition, or software crash at some point during a multi-week run is close to a statistical certainty, not a rare event. Checkpointing — periodically saving enough state to resume training from a recent point rather than from scratch — is what makes long-running training economically survivable. Without it, a single failure late in a training run could mean discarding weeks of compute spend.

What a checkpoint actually needs to contain

A correct checkpoint needs more than just model weights. To resume training *exactly* as if it had never stopped, you generally need: the model parameters, the optimizer state (e.g., Adam's momentum and variance buffers — resuming with fresh optimizer state can cause a destabilizing loss spike), the learning rate scheduler's state, the current step/epoch count, and the data loader's position in the dataset (so already-seen examples aren't repeated or skipped incorrectly on resume). Omitting any of these produces a 'resume' that silently behaves differently than an uninterrupted run — sometimes subtly, sometimes catastrophically.

checkpoint_save_load.py

python

import torch

def save_checkpoint(path, model, optimizer, scheduler, step, dataloader_state):
    torch.save({
        "model_state_dict": model.state_dict(),
        "optimizer_state_dict": optimizer.state_dict(),
        "scheduler_state_dict": scheduler.state_dict(),
        "step": step,
        "dataloader_state": dataloader_state,   # e.g. sampler position / shuffled indices
    }, path)

def load_checkpoint(path, model, optimizer, scheduler):
    ckpt = torch.load(path, map_location="cpu")
    model.load_state_dict(ckpt["model_state_dict"])
    optimizer.load_state_dict(ckpt["optimizer_state_dict"])
    scheduler.load_state_dict(ckpt["scheduler_state_dict"])
    return ckpt["step"], ckpt["dataloader_state"]

# Save on a fixed interval, plus always on graceful shutdown signals (e.g. SIGTERM)
if step % checkpoint_every_n_steps == 0:
    save_checkpoint(f"ckpt_step_{step}.pt", model, optimizer, scheduler, step, dataloader.state())

The frequency trade-off

Checkpointing more often reduces how much compute is lost on failure, but checkpoint saves aren't free — writing a large model's full state (potentially hundreds of gigabytes for a frontier model, including optimizer state) to persistent storage takes real time and I/O bandwidth, and can stall training if done naively and synchronously. Teams tune checkpoint frequency against observed hardware failure rates for their specific cluster: a reasonable target is often choosing an interval such that expected compute lost to failure roughly balances the overhead of checkpointing itself, though in practice most large runs err toward more frequent checkpoints given how costly losing significant training time is.

Checkpointing strategy options

Strategy	How it works	Overhead	Best for
Synchronous full checkpoint	Training pauses while full state is written to disk	High — training stalls during save	Simpler pipelines, infrequent saves
Asynchronous checkpoint	State is copied to CPU/background process; training continues	Low — minimal training stall	Frequent checkpointing on large-scale runs
Distributed/sharded checkpoint	Each device saves only its own shard of state (pairs with ZeRO/FSDP)	Scales well with cluster size	Very large models with sharded parallelism
Incremental/delta checkpoint	Saves only changes since last checkpoint	Lower storage cost per save	Very frequent checkpointing needs

Elastic/fault-tolerant training needs more than just saving state

Resuming after a failure in a distributed job isn't just about having a saved checkpoint — the training framework also needs to detect the failure, potentially re-provision replacement hardware, re-establish the distributed communication topology (process groups, rank assignments), and reload the checkpoint onto possibly different physical devices than before. Frameworks like PyTorch's elastic training and tools built around it exist specifically to automate this recovery sequence so a human doesn't have to manually restart a multi-thousand-GPU job by hand.

Checkpoints double as experiment artifacts, not just insurance

Beyond fault tolerance, periodic checkpoints serve a second purpose: they let researchers evaluate the model's capability at various points during training, compare learning dynamics across runs, or roll back to an earlier checkpoint if a later stage of training turns out to have degraded quality (for instance, due to a data quality issue introduced partway through). This dual purpose is part of why checkpoint retention policies often keep more history than strict fault-tolerance alone would require.

What's next?

Checkpointing is an operational safeguard for training that's otherwise working as intended. A more conceptual question lurks underneath all of this engineering: what does the geometry of the loss function that training is navigating actually look like, and why does gradient descent manage to find good solutions in such an enormous, high-dimensional space at all.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →