The A/B Testing Statistics Most Teams Skip Until It Bites Them

I've reviewed enough client experimentation programs now to notice a pattern: the testing tool is set up correctly, the hypothesis is reasonable, the test runs cleanly — and the statistics underneath the headline result are almost never checked by anyone, because the tool shows a green checkmark and a percentage, and that feels like enough.

The checkmark usually just means the observed difference crossed a significance threshold at the moment someone looked. It says nothing about whether the sample size was adequate for the effect size being claimed, whether multiple metrics were tested until one happened to turn green, or whether the test had been peeked at and stopped the moment it looked good.

The specific gaps that cause real damage

• No pre-registered minimum sample size, so tests get called the moment they look significant rather than the moment they're actually powered to detect the claimed effect
• Testing a dozen secondary metrics and reporting whichever one crossed significance, without correcting for the fact that testing enough metrics guarantees a false positive eventually
• Stopping a test early because it 'looked done,' which dramatically inflates the false positive rate compared to a test run to its planned duration
• Treating a p-value just under the threshold as meaningfully different from one just over it, when the actual underlying uncertainty barely changed

A green checkmark from a testing tool tells you a threshold was crossed. It doesn't tell you whether the result means anything.

What a basic fix looks like

None of this requires a statistics degree to fix. It requires calculating a sample size before the test starts, committing to a stopping point in advance, and accepting that a barely-significant result on a single primary metric deserves far less confidence than teams usually grant it. I've watched a client roll out a 'winning' pricing page change based on a test that was stopped three days early because it looked good — the actual rollout showed no measurable lift, because the early stop had caught statistical noise, not a real effect.

The statistics aren't the interesting part of experimentation to most teams, which is exactly why they get skipped. But skipping them doesn't make the uncertainty go away — it just means you find out about it after you've already shipped a decision based on it.