An A/B Test 'Win' That Was Actually a False Positive

The test ran for nine days, hit statistical significance on day seven, and showed a 14% lift in checkout completion for the new variant. We called it, shipped it to 100% of traffic, and told the client it was a win. Six weeks later the checkout completion rate for the full population was flat against the pre-test baseline. The 14% lift had been real in the test and completely absent at scale.

What actually happened

We'd stopped the test the moment it crossed significance instead of running it for the pre-committed sample size and duration. Day seven happened to fall during a weekend, when the traffic mix skewed toward a segment that was already more likely to convert regardless of variant. The test wasn't wrong about those nine days — it was wrong about generalizing from them, because we never let it run through a full weekly cycle to average out the mix.

A statistically significant result on a non-representative sample isn't a smaller win. It's not a win at all, and the math won't tell you the difference.

What we changed after this

• Pre-register the sample size and minimum duration before launch, and treat early significance as information, not a stop signal.
• Always run tests through at least one full weekly cycle, regardless of when significance is hit, to avoid day-of-week traffic mix skew.
• Segment the result by traffic source and device before declaring a win — a global lift that's actually concentrated in one segment is a different, smaller finding.
• Run a holdout group post-launch for any test result that materially changes a core flow, so a false positive gets caught before it's treated as settled fact.

The expensive part of this mistake wasn't the lost development time on the new variant — it was the two months the client spent reporting a 14% improvement that didn't exist, before the holdout group we eventually added caught the discrepancy. Peeking at a test early doesn't just risk a wrong call. It risks a wrong call that gets repeated in a board deck before anyone double-checks it.