A Power Analysis That Prevented a Wasted Quarter of Testing

A client came to me wanting to test a redesigned checkout flow against the existing one, expecting a clean readout in about three weeks. Their baseline conversion rate was 2.1%, they were expecting the new flow to lift it to maybe 2.3%, and they had roughly 4,000 checkout sessions a week split across both arms. Nobody had run the numbers on whether that was enough traffic to detect anything.

The math nobody wanted to do first

A power analysis for a two-proportion test at that baseline and that minimum detectable effect, with standard 80% power and a 95% confidence level, needed roughly 60,000 sessions per arm. At 2,000 sessions a week per arm, that's thirty weeks — about seven months, not the three they'd budgeted. Nobody had sat down and done this calculation before scheduling the test, because 'let's just run it and see' feels like progress and a power analysis feels like paperwork.

An underpowered test doesn't fail loudly. It just runs forever and tells you nothing, while everyone upstream assumes it's working.

What we did instead

We had two real options: accept a much longer test window, or redesign the test to detect a larger effect. We went with the second — instead of testing the full redesigned flow against the baseline, we isolated the single highest-hypothesis change (removing a redundant form field) and tested that in isolation, where the team's internal data suggested a larger effect size closer to a full percentage point. That cut required sample size to about 12,000 sessions per arm, which was achievable in six weeks instead of thirty.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.021, 0.023)  # baseline vs hoped-for lift
analysis = NormalIndPower()
n = analysis.solve_power(effect_size=effect, alpha=0.05, power=0.8)
print(n)  # ~60,000 per arm

• Run the power analysis before scheduling the test, not after it's already underway and the numbers aren't moving.
• If the math says the test needs more traffic than you have, don't run it anyway and hope — redesign for a bigger effect or a smaller scope.
• Communicate sample size requirements to stakeholders in weeks, not percentages — nobody internalizes 'n=60,000' but everyone understands 'this takes seven months.'

The redesigned test hit significance in five and a half weeks with a real, defensible lift. The original plan would have burned a whole quarter on a test that was never going to resolve either way — and worse, somebody would have eventually just called it on vibes, which is exactly the outcome a power analysis exists to prevent.