A/B Test Power Calculations for ML Teams
Why ML teams can't skip power calculations
When an ML team ships a new ranking model, a new recommendation algorithm, or a new bandit policy, the honest way to know if it actually helped is an A/B test against the current production system. But a test with too few users will fail to detect a real improvement (a false negative, or Type II error), while a test run carelessly can report a win that's actually noise (a Type I error). A power calculation tells you, before you launch, how many users or how many days you need to reliably tell a real effect apart from random variation — so you don't ship on noise, and don't kill a genuinely better model because the test was underpowered.
The four ingredients of a power calculation
Every power calculation balances four quantities, and fixing any three determines the fourth:
1. Significance level ($\alpha$) — the Type I error rate you'll tolerate, conventionally 0.05.
2. Power ($1-\beta$) — the probability of detecting the effect if it's real, conventionally 0.8.
3. Minimum detectable effect (MDE) — the smallest effect size you care about catching, e.g. a 2% relative lift in click-through rate.
4. Sample size ($n$) — the number of units (users, sessions, impressions) per arm.
For a two-proportion test comparing conversion rates $p_1$ and $p_2$, the standard approximate sample size per arm is:
$n = \frac{(z_{\alpha/2} + z_\beta)^2 \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_1 - p_2)^2}$
where $z_{\alpha/2} \approx 1.96$ for $\alpha=0.05$ and $z_\beta \approx 0.84$ for 80% power.
import numpy as np
from scipy import stats
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
baseline_rate = 0.05 # current model's conversion/CTR
mde_relative = 0.10 # smallest relative lift worth detecting (10%)
treatment_rate = baseline_rate * (1 + mde_relative)
effect_size = proportion_effectsize(treatment_rate, baseline_rate)
power_analysis = NormalIndPower()
n_per_arm = power_analysis.solve_power(
effect_size=effect_size,
alpha=0.05,
power=0.80,
ratio=1.0,
alternative="two-sided",
)
print(f"Required sample size per arm: {n_per_arm:.0f}")
print(f"Total users needed: {2 * n_per_arm:.0f}")
# Sanity check with the closed-form formula
z_alpha = stats.norm.ppf(1 - 0.05 / 2)
z_beta = stats.norm.ppf(0.80)
p1, p2 = baseline_rate, treatment_rate
n_closed_form = ((z_alpha + z_beta) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2))) / (p1 - p2) ** 2
print(f"Closed-form estimate: {n_closed_form:.0f}")
Underpowered ML experiments are the norm, not the exception
Variance reduction: getting more power from the same traffic
Rather than always running tests longer, ML teams can shrink the variance of the metric itself to detect smaller effects with the same sample size. CUPED (Controlled-experiment Using Pre-Experiment Data, Microsoft 2013) uses each user's pre-experiment behavior as a covariate to reduce residual variance: $Y_{adj} = Y - \theta(X - \bar{X})$, where $X$ is the pre-period covariate and $\theta$ is chosen to minimize variance (typically the OLS coefficient of $Y$ on $X$). This can cut required sample size by 30-50% in metrics with strong user-level persistence, such as revenue or engagement, without touching the actual experiment mechanics. Stratified sampling and ratio metrics (e.g. per-session rather than per-user) are other common variance-reduction levers.
Metric choice and multiple comparisons
ML experiments often track dozens of metrics (latency, multiple engagement signals, downstream revenue, fairness slices). Running many simultaneous significance tests inflates the overall false positive rate — with 20 independent metrics tested at $\alpha=0.05$, the chance of at least one false positive is roughly $1-(0.95)^{20} \approx 64\%$. Pre-register a single primary metric the power calculation is built around, treat everything else as secondary/guardrail metrics interpreted more cautiously, and apply a correction (Bonferroni, or better, Benjamini-Hochberg for false discovery rate control) when you do need to interpret many secondary metrics formally.
Power calculations also bound how long to run sequential tests
What's Next?
Power calculations assume you're running a single, static A/B split for a fixed horizon. Many ML systems — recommendation, ads, and pricing in particular — instead adapt traffic allocation in real time toward whichever arm is performing best, trading some statistical cleanliness for faster exploitation of good options. Understanding how that adaptive allocation works, and the tradeoffs it makes against a classic fixed-horizon test, is the natural next step.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.