Holdout Group Testing for Marketing
From One-Off Tests to a Standing Program
A single incrementality test answers one question about one campaign at one point in time. A holdout group *program* is a standing operational practice: a persistent, randomly assigned segment of your audience that's permanently or semi-permanently excluded from some or all marketing touchpoints, maintained continuously so you always have a live, current baseline to measure true incremental lift against — rather than having to design and justify a new ad-hoc test every time you want to know if marketing is working.
Designing the Holdout Allocation
The central design decision is holdout size and scope. Too small a holdout (1-2% of audience) makes it hard to detect lift with statistical confidence on anything but the largest campaigns; too large a holdout (20%+) represents meaningful forgone revenue if the marketing genuinely works. Most mature programs settle around 5-10% for a global, always-on holdout, sized via the same power analysis principles as any incrementality test, calibrated to the smallest effect size you actually care about being able to detect reliably.
-- Deterministic, stable holdout assignment: hash the user ID so the
-- same user always lands in the same bucket across campaigns and over time,
-- rather than re-randomizing per campaign (which would contaminate the holdout).
SELECT
user_id,
CASE
WHEN MOD(ABS(FARM_FINGERPRINT(user_id)), 100) < 10 THEN 'holdout'
ELSE 'marketable'
END AS holdout_bucket
FROM `project.crm.users`
WHERE consent_status = 'opted_in'Stability of Assignment Is What Makes a Holdout a Holdout
What You Can Measure With a Standing Holdout
Beyond lift for any single campaign, a maintained holdout lets you answer questions a one-off test can't: cumulative incremental value of an entire marketing program over a quarter or year, marginal incrementality of adding a new channel to an already-marketed audience, and — critically — whether incrementality is declining over time as a channel or audience saturates, something only visible with continuous measurement rather than a point-in-time snapshot.
Organizational and Political Realities
Holdout programs face a recurring internal pressure: every individual campaign owner wants their specific campaign excluded from the holdout exclusion ('just this once, for this high-value launch'), and granting those exceptions repeatedly erodes the holdout's integrity until it no longer represents a clean untouched baseline. The program needs clear, centrally enforced governance — ideally owned by an analytics or growth team independent of individual campaign owners — with a high bar for exceptions and transparent tracking of when and why any exception was granted.
CRM and Ad Platform Suppression Must Actually Match
Measuring the Holdout Itself: Statistical Discipline Over Time
As a holdout runs for months or years rather than weeks, standard considerations compound: multiple comparisons across many campaigns measured against the same holdout inflate false-positive risk and need correction (e.g., controlling false discovery rate rather than treating each campaign's test as independent), and the holdout composition itself can drift from the overall population if churn or opt-outs aren't random with respect to holdout status — periodically re-validate that the holdout group remains demographically and behaviorally representative of the marketable population.
What's Next?
Display and programmatic advertising have their own specialized variant of this holdout concept, built directly into ad serving — the next lesson covers Ghost Ads and PSA testing specifically.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.