Traditional software has a reproducibility problem solved reasonably well by version control and containerization: pin the code and the dependencies, and you get the same behavior every time. ML adds at least three more sources of variation on top of that — randomness baked into the training process itself, data that changes over time even when nobody edits a file, and hardware/library versions that can silently change numerical results. A model that scored 94% accuracy last Tuesday can genuinely fail to reproduce that number next Tuesday, on the same code, for reasons that have nothing to do with a bug.

Randomness is everywhere in a training run

Random number generators control far more of a training run than people usually track: weight initialization, the order data is shuffled in, which units get dropped by dropout, data augmentation choices, and even which GPU kernel implementation gets selected (some are intentionally non-deterministic for speed). Fixing a single 'random seed' variable is necessary but not sufficient — a reproducible run needs every one of these sources seeded consistently, and on GPUs, some low-level operations (certain parallel reduction orders, for instance) are non-deterministic by default regardless of seeding unless you explicitly request deterministic algorithms, usually at some performance cost.

python

Note

Chasing exact, bitwise-identical results across different hardware or library versions is often not worth the engineering cost or the performance penalty of forcing fully deterministic (and often slower) operations. The practical goal for most teams is statistical reproducibility: a rerun should land within a known, acceptable tolerance of the original result, and the gap should be explainable (different random seed, different hardware) rather than mysterious. Save true bitwise determinism for cases where it's actually required — regulated industries, debugging a suspected bug, or scientific publication.

Environment drift: the quiet killer

A training script that worked perfectly six months ago can produce different results today because an underlying library updated its default behavior, a CUDA driver changed how certain operations round floating-point numbers, or a transitive dependency silently bumped a version. 'It worked on my machine' becomes 'it worked on my machine, on that day, with those exact library versions' — which is a much narrower and more fragile claim than most teams realize until they try to rerun something old. Pinning exact dependency versions (not just major versions), and ideally freezing the entire environment in a container image, is the standard defense, but it has to be done deliberately — a requirements file with loose version ranges gives you none of this protection.

Data versioning: the input that changes without anyone editing a file

Code and environment can be pinned with familiar tools, but training data is often the most under-versioned part of the pipeline. A training dataset pulled live from a production database or a feature store can change underneath you — rows get added, labels get corrected, upstream data gets backfilled — such that rerunning 'the same' training job next month trains on genuinely different data without a single line of code changing. Reproducible ML pipelines snapshot or version the exact data used for a given training run (a dataset hash, a frozen snapshot, or a data-versioning tool), so 'what data produced this model' is answerable months later with certainty rather than a guess.

Sources of non-reproducibility and their fixes

Source of variation	Why it breaks reproducibility	Standard fix
Random initialization/shuffling/dropout	Different runs sample different randomness	Seed every RNG, not just one global seed
Non-deterministic GPU kernels	Some fast kernels don't guarantee operation order	Request deterministic algorithms (accepts a speed cost)
Library/driver version drift	Defaults and numerics change across versions	Pin exact versions; freeze environment in a container
Changing underlying data	Data changes without any code change	Snapshot/version the exact dataset used per run
Undocumented manual steps	A human did something not captured in code	Fully scripted pipelines; no manual intervention steps

Note

The practical backbone of ML reproducibility is an experiment tracking system that automatically logs, for every run: the exact code commit, the full set of hyperparameters, the data snapshot reference, the environment/container identifier, the random seeds, and the resulting metrics and model artifact. Done well, answering 'what exactly produced this model in production' becomes a lookup rather than an archaeology project — and done poorly (or not at all), that same question can consume days of a team's time during an incident.

What's next

One of the most common silent reproducibility failures is a mismatch between how a feature is computed at training time versus serving time. The next lesson, Feature Stores Explained, covers the architecture built specifically to eliminate that mismatch.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →