Before running an A/B test or an experiment comparing two models, there's a question that matters more than almost anything you'll do during the analysis: how much data do you actually need? Run with too little data, and a real effect can hide in noise, leading you to wrongly conclude 'no difference.' Collect far more than necessary, and you've wasted time and resources, or kept users in a suboptimal experience longer than needed.

Power analysis answers this by connecting four quantities — effect size, sample size, significance level, and statistical power — such that if you fix any three, the fourth is determined. Most commonly, you fix the first three and solve for the sample size you need.

Type I and Type II errors: the two ways a test can be wrong

A Type I error (false positive) is concluding there's an effect when there isn't one — rejecting a true null hypothesis. Its probability is α (alpha), the significance level you choose, conventionally 0.05.

A Type II error (false negative) is failing to detect an effect that actually exists — failing to reject a false null hypothesis. Its probability is β (beta). Statistical power is 1 − β: the probability your test correctly detects a real effect of a given size. Conventionally, experiments target 80% power, meaning a 20% chance of missing a real effect.

These two error types trade against each other through sample size: for a fixed effect size, the only way to lower both α and β simultaneously is to collect more data.

Note

Significance level (α), power (1 − β), effect size, and sample size (n) are mathematically linked — specify any three and the fourth is fixed. Power analysis almost always fixes α (0.05) and power (0.80) by convention, specifies the smallest effect size worth detecting, and solves for n. Running the calculation after the fact with the observed effect size (post-hoc power analysis) is a well-known statistical malpractice — it just restates how significant your result already was.

Effect size: the piece you have to decide in advance

Effect size is the magnitude of the difference you actually care about detecting — not the magnitude you expect to find, but the smallest difference that would be practically meaningful. A one-tenth of a percent lift in conversion rate might be statistically detectable with enough data, but is it worth running the experiment for? Deciding the minimum detectable effect (MDE) you care about, before collecting data, is what makes the rest of the calculation meaningful.

For comparing two proportions (like conversion rates), effect size is commonly expressed with Cohen's h; for comparing two means, with Cohen's d (the standardized mean difference). Smaller effect sizes require dramatically more data to detect reliably — this is the single biggest lever in the whole calculation.

python

How much sample size changes with each lever

Change	Effect on required sample size	Why
Halve the effect size (MDE)	Roughly 4x more samples needed	Required n scales with 1/(effect size)^2
Tighten α from 0.05 to 0.01	Noticeably more samples	A stricter significance threshold demands more evidence to clear the bar
Raise power from 80% to 95%	Meaningfully more samples	Detecting the effect more reliably requires a larger margin over noise
Switch one-sided to two-sided test	Somewhat more samples	Two-sided splits significance across both tails of the distribution
Reduce outcome variance (better instrumentation)	Fewer samples needed	Lower noise makes a given effect size easier to detect

Note

An underpowered study that does happen to reach statistical significance tends to overestimate the true effect size — a phenomenon sometimes called the 'winner's curse' in experimentation. If you only ever see significant results from a noisy, underpowered process, the published or acted-upon effects will systematically look larger than reality. This is a major contributor to effects that look strong in an initial test and shrink or vanish on a larger follow-up.

Sample size in ML model evaluation, not just A/B tests

The same logic applies directly to deciding how large a labeled test set needs to be before you trust a comparison between two models. If you want to reliably detect a 1-point accuracy difference between two classifiers, a power analysis (treating it as a comparison of two proportions) will tell you that you likely need a test set in the thousands, not the hundreds — which is exactly why small benchmark test sets routinely produce leaderboard rankings that don't replicate.

What's next

Power analysis tells you how much data you need before you run a test. But before trusting any A/B test result at all, you need to know the testing system itself is sound — that it doesn't report false positives when there's genuinely no difference. The next lesson, A/A Testing & Experiment Validity Checks, covers how to verify that.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →