An A/B test's entire value rests on an assumption that's easy to overlook: that when there is truly no difference between two groups, the test reports 'no significant difference' roughly 95% of the time (at α = 0.05) — not more often, not less. If your experimentation pipeline has a bug — broken randomization, a logging error, a miscalibrated p-value calculation — that assumption can quietly fail, and every A/B test you run on top of it becomes unreliable in a way that's invisible from the results alone.

An A/A test checks this directly: split traffic into two groups using your real experimentation system, show both groups the *exact same* experience, and run your normal statistical analysis. Since there is no actual treatment difference, a healthy system should report a statistically significant difference only about 5% of the time, matching your chosen significance level.

Note

In a real A/B test, you never know for certain whether the null hypothesis (no difference) is true — that's what you're testing. In an A/A test, you know it's true by construction, because both groups get identical treatment. This makes A/A tests one of the only places in experimentation where you can directly verify your statistical machinery against ground truth rather than trusting it on faith.

What A/A tests catch

Inflated false positive rates. If repeated A/A tests show significant results 15% of the time instead of 5%, something in your pipeline — often correlated data points being treated as independent, or a bug in variance estimation — is making your test overconfident. Every real A/B test run on the same system will report more false wins than it should.

Sample ratio mismatch (SRM). If you intend a 50/50 split but consistently observe something like 52/48, your randomization isn't working as designed — commonly caused by bot traffic hitting one variant disproportionately, caching serving one variant more often, or a bucketing bug. SRM invalidates the entire experiment, independent of what the metric results show, because it means the two groups weren't actually comparable to begin with.

Metric instrumentation bugs. If an A/A test on a specific metric consistently shows one 'variant' outperforming the other even though both show the literal same experience, the metric's logging or computation pipeline — not the experiment — is broken.

python

Running an A/A test as a calibration check

The standard protocol: run many simulated A/A tests (either genuinely in production with a true no-op split, or by resampling historical data and randomly splitting it into two 'fake' groups), and track how often each one reports a significant result. Under a correctly calibrated system at α = 0.05, roughly 5% of these simulated tests should cross the significance threshold — no more, no less. This is directly checkable by simulation even without running a live A/A test in production.

python

Common experiment validity issues and how to catch them

Issue	Symptom	Detection method	Typical cause
Inflated false positive rate	A/A tests show significance far more than α of the time	Repeated A/A simulation, calibration check	Non-independent observations, variance miscalculation
Sample ratio mismatch	Observed traffic split deviates from intended allocation	Chi-squared goodness-of-fit test on bucket counts	Bot traffic, caching, bucketing bugs
Novelty effect	Metric shift fades over the course of the experiment	Plot the metric difference over time, not just the final aggregate	Users reacting to change itself, not the treatment
Metric instrumentation bug	Metric differs between identical A/A arms	A/A test specifically targeting that metric	Logging error, race condition, inconsistent event firing
Peeking / early stopping bias	Significance found early, then disappears with more data	Pre-register sample size, use sequential testing methods if stopping early	Checking p-values repeatedly and stopping at first 'significant' result

Note

Checking an experiment's p-value every day and stopping as soon as it crosses 0.05 inflates the real false positive rate far above the nominal 5%, because you're effectively giving yourself many chances to get a lucky significant result. If you need to monitor results as they accumulate, use a sequential testing method (such as sequential probability ratio tests or always-valid confidence sequences) designed to control the false positive rate under continuous monitoring — don't apply a fixed-sample-size test and peek anyway.

What's next

A/A testing validates the measurement system around a model or product decision. The next lessons shift from experiment statistics into the mechanics of training neural networks themselves — starting with the Adam optimizer, the algorithm most modern deep learning models actually use to turn gradients into weight updates.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →