Running an Incrementality Test on a Budget That's Too Small for It

A client spending around $40k a month on paid social wanted to know if it was doing anything. The textbook answer is that $40k a month is too small to run a clean incrementality test — not enough statistical power to detect a modest lift with confidence. We ran it anyway, because the alternative was spending another year not knowing.

The setup

We split comparable geo regions into test and holdout, paused spend in the holdout regions for six weeks, and compared revenue trends against the test regions using a pre-period baseline to control for seasonality. The expected outcome, going in, was an inconclusive result with a wide confidence interval — which still would have been useful information, just not a headline number.

An underpowered test that returns 'we can't tell' is still more honest than a last-click dashboard that confidently tells you the wrong number.

What actually happened

The holdout regions showed almost no revenue drop at all — nowhere near what last-click attribution had been crediting to that channel. The confidence interval was wide, as predicted, but it was wide around a number close to zero, not wide around some meaningfully positive number. That's a different kind of result than 'inconclusive': it's not precise, but it rules out the channel being the growth driver the dashboard claimed it was.

• A small-budget test can't give you a precise lift percentage, but it can often rule out a channel being as incremental as attribution claims.
• Six weeks was the minimum to smooth out week-to-week noise; four weeks would have been too short to trust.
• Geo holdouts are cheaper and less politically fraught than full audience holdouts for a first test.
• The client's next move wasn't 'kill the channel' — it was 'cut the budget in half and re-test', which is the right level of confidence for this data.

Don't let 'not statistically powerful enough for a precise answer' talk you out of running the test. A directionally confident 'probably not working' from a small test beats a precisely wrong number from attribution you've already stopped trusting.