We Miscalculated a Sample Size and Called a Test Early, a Postmortem

The sample size calculator we used asked for baseline conversion rate, minimum detectable effect, and significance level. We filled it in using last quarter's conversion rate as the baseline. The problem: that quarter had an unusually high-converting cohort because of a promo, and the calculator spat out a sample size that was too small for the site's normal traffic pattern.

Nine days in, the test showed a 12% lift on the new pricing page with a p-value comfortably under 0.05. We called it, shipped the winner, and moved on to the next test. Three weeks later someone on the data team re-ran the numbers out of curiosity and found the lift had shrunk to under 2% once the full, correctly-sized sample had actually run.

How the mistake surfaced

It wasn't caught by a review process — we didn't have one for sample size validation at the time. It surfaced because someone noticed the post-launch revenue numbers didn't match the 12% projection, went looking for why, and found the test had been statistically underpowered from the start. Significance at day nine with a too-small required sample is exactly the kind of false positive you'd expect from early variance, not a real effect.

A test calling itself significant on a wrong sample size isn't an early win, it's noise wearing a confidence interval.

Correct baseline: use a trailing 90-day average, not last quarter alone
Recompute sample size if traffic mix changes mid-test
Don't trust significance before the pre-committed sample size is hit
Log the calculator inputs alongside the test, not just the output

What changed after

We now require the sample size calculation inputs to be reviewed by a second person before a test launches, and the test dashboard shows percentage of required sample reached, not just "significant: yes/no". It's a small process addition, but it would have caught this one in about thirty seconds.