Classical significance testing assumes you fix a sample size in advance, collect exactly that much data, and compute a p-value exactly once. In practice, almost nobody does this — teams check the live results dashboard daily (or hourly) and feel the pull to stop the moment p < 0.05 appears. This behavior, called optional stopping or peeking, is one of the most common and most damaging statistical errors in applied marketing experimentation, because each additional look at the data is, in effect, another chance for random noise to cross the significance threshold even when there's truly no effect.

How bad the inflation actually gets

The math here is counterintuitive to most people's intuition: if you peek at a test's results five times over its run and stop the first time p < 0.05 appears, the *actual* probability of a false positive (concluding there's an effect when there isn't one) isn't 5% — it's closer to 14-15%. Peek continuously and let yourself stop at any point, and the true false-positive rate approaches 100% given enough time, because a random walk will eventually cross any fixed threshold if you wait long enough. This is sometimes visualized as p-value trajectories randomly wandering above and below the 0.05 line throughout a test's run — and 'stop the first time it dips below' systematically selects for noise, not signal.

peeking_inflation_illustration.txt

text

TRUE EFFECT: ZERO (A and B are identical; any "win" is pure noise)

Day 1:  p = 0.41   (no action - not significant)
Day 3:  p = 0.22   (no action)
Day 5:  p = 0.09   (no action)
Day 7:  p = 0.04   <- "SIGNIFICANT! Ship it!" -- but this is a false positive,
                       and with 5 looks at the data, false-positive rate here
                       is closer to 14%, not the nominal 5% a single-look test assumes.
Day 9:  p = 0.11   (if they'd kept watching, it would have bounced back up)
Day 14: p = 0.38   (converged back toward "no effect," as it should)

Early 'wins' are disproportionately noise

A test that shows significance very early, on a small fraction of its planned sample, is statistically more likely to be a noise-driven false positive than a test that reaches the same significance level only after running its full planned duration. 'It hit significance on day 3 of a planned 3-week test' should raise suspicion, not excitement — treat an early apparent win as a reason to keep running the test to its pre-committed sample size, not a reason to stop.

Sequential testing methods: built for legitimate early stopping

The fix isn't 'never check results early' — it's using a statistical method explicitly designed to allow valid peeking. Sequential testing methods (group sequential designs, the mSPRT — mixture sequential probability ratio test — used by some modern testing platforms) adjust the significance threshold dynamically as more looks are taken, specifically correcting for the inflation that naive repeated peeking causes. These methods let a test stop early and legitimately when the evidence is strong enough, without the inflated false-positive rate of checking a standard fixed-sample p-value test repeatedly.

Fixed-horizon vs. sequential testing

Dimension	Fixed-horizon (classical)	Sequential (mSPRT, group sequential)
Valid to check early and stop	No — invalidates the stated alpha	Yes — built to allow it
Requires pre-committing sample size	Yes, strictly	Less strictly — adapts as data comes in
Common tooling support	Universal (any basic stats library)	Growing (Optimizely 'Stats Engine', some VWO modes)
Risk if misapplied	Peeking inflates false positives silently	Lower, but still requires correct implementation

If your tool doesn't support sequential testing, pre-commit and look away

The cheapest, most reliable fix available to any team without access to a sequential-testing-aware platform is simply discipline: set the sample size and duration before launch, write it down somewhere visible, and don't act on interim numbers regardless of how they look until that pre-committed point is reached. It's less elegant than a proper sequential method, but it's far better than naive peeking and requires no special statistical tooling.

What's Next?

Getting the statistics right on individual tests is foundational, but it only pays off at scale if there's a system for running many tests reliably, consistently, and with shared infrastructure — which is what a mature experimentation platform is built to provide.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →