Statistical power is the probability that a test correctly detects a true effect, given that the effect actually exists at the size you're assuming. A test with 80% power (the conventional minimum target) means that if the true effect is exactly the size you designed for, you'll correctly detect it 80% of the time across repeated trials — and miss it (a false negative, or Type II error) the other 20% of the time. Power is a function of four interacting inputs: baseline conversion rate, minimum detectable effect (MDE), sample size, and significance threshold (alpha). Fix any three and the fourth is determined — which is exactly how a sample size calculator works.

Why most underpowered marketing tests fail silently

The dangerous failure mode isn't a test that runs too long — it's a test that's called 'inconclusive' or, worse, 'no significant difference, ship the cheaper option' after running for two weeks on insufficient traffic. An underpowered test that shows no significant result tells you almost nothing about whether a real effect exists; it's equally consistent with 'there's no effect' and 'there's a real, meaningful effect but this test didn't have enough traffic to detect it reliably.' Teams that don't run a power calculation before launching a test routinely mistake the second case for the first, and discard genuinely winning ideas because the test that should have proven them was never adequately powered to do so.

sample-size-estimate.ts

typescript

// Simplified two-proportion sample size estimate (per variant),
// for a two-sided test at alpha = 0.05, power = 0.80.
// Not a substitute for a proper calculator, but shows the mechanics.
function estimateSampleSizePerVariant(
  baselineRate: number,  // e.g. 0.03 for a 3% conversion rate
  minimumDetectableEffect: number // relative, e.g. 0.10 for a 10% relative lift
): number {
  const p1 = baselineRate;
  const p2 = baselineRate * (1 + minimumDetectableEffect);
  const pooledP = (p1 + p2) / 2;

  // z-scores for alpha=0.05 two-sided (1.96) and power=0.80 (0.84)
  const zAlpha = 1.96;
  const zBeta = 0.84;

  const numerator =
    Math.pow(zAlpha * Math.sqrt(2 * pooledP * (1 - pooledP)) +
      zBeta * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2)), 2);
  const denominator = Math.pow(p2 - p1, 2);

  return Math.ceil(numerator / denominator);
}

// Example: 3% baseline, hoping to detect a 10% relative lift (3.0% -> 3.3%)
// requires roughly 28,700 visitors PER VARIANT - a number that surprises
// most people planning a two-week test on moderate traffic.
console.log(estimateSampleSizePerVariant(0.03, 0.10));

Small baseline rates require dramatically more traffic

The lower the baseline conversion rate, the more total traffic is needed to detect the same relative effect size, because variance scales with p(1-p) while the absolute gap you're trying to detect shrinks. Detecting a 10% relative lift on a 3% baseline conversion rate takes far more traffic than detecting the same 10% relative lift on a 15% baseline rate — a frequently painful realization for teams testing low-funnel, rare-conversion events like free-to-paid upgrades.

What changes required sample size

Input	Effect of increasing it	Practical lever
Minimum detectable effect (MDE)	Smaller MDE requires exponentially more sample	Only commit to detecting lifts worth acting on; don't chase tiny effects
Baseline conversion rate	Lower baseline requires more sample for the same relative effect	Test higher up the funnel where events are more frequent, if the business question allows it
Significance threshold (alpha)	Stricter alpha (e.g. 0.01 vs 0.05) requires more sample	Use 0.05 unless the decision stakes justify stricter
Desired power	Higher power (e.g. 90% vs 80%) requires more sample	80% is the standard default; raise only for high-stakes, hard-to-rerun tests

Power calculations belong before the test launches, not after

The discipline that actually prevents wasted tests is running the sample size calculation at design time and using it to set an explicit, pre-committed test duration — then holding to that duration regardless of how the interim numbers look. A power calculation run after the fact, to justify why an already-finished test's result should be trusted, defeats the entire purpose.

What's Next?

Classical power calculations assume a fixed sample size decided in advance — but teams that peek at results early and stop as soon as they look significant run into a related and even more common statistical trap worth understanding on its own terms.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →