Incrementality Testing Design

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The Question Incrementality Testing Actually Answers

Every attribution model discussed elsewhere in this track — last-touch, data-driven, Shapley, Markov — answers 'which touchpoints correlate with conversion.' Incrementality testing answers a fundamentally different and more valuable question: 'how many of these conversions would not have happened without this specific marketing activity?' This requires a true causal design — a treatment group exposed to the activity and a comparable control group that isn't, with the difference in outcomes isolating the activity's actual incremental effect, independent of any attribution model's assumptions.

Geo-Based Holdout Testing

The most common and practical incrementality design for channels where individual-level randomization is impractical (TV, broad paid social, display) is a geo holdout: split comparable geographic markets into treatment (campaign runs normally) and control (campaign is paused or suppressed entirely), then compare the aggregate outcome lift in treatment geos against control geos over the test period. Geo selection matters enormously — treatment and control geos need to be genuinely comparable in baseline demand, seasonality, and demographic composition, usually validated by checking historical outcome correlation between candidate geo pairs before the test starts, not just picking geos that seem similar on the surface.

geo_holdout_analysis.sql

sql

-- Pre-test validation: check historical correlation between candidate
-- treatment and control geo pairs before committing to the split.
WITH geo_weekly_sales AS (
  SELECT
    geo_market,
    DATE_TRUNC(order_date, WEEK) AS week,
    SUM(revenue) AS weekly_revenue
  FROM `project.sales.orders`
  WHERE order_date BETWEEN '2025-01-01' AND '2025-12-31'
  GROUP BY 1, 2
)
SELECT
  a.geo_market AS geo_a,
  b.geo_market AS geo_b,
  CORR(a.weekly_revenue, b.weekly_revenue) AS correlation
FROM geo_weekly_sales a
JOIN geo_weekly_sales b ON a.week = b.week AND a.geo_market < b.geo_market
GROUP BY 1, 2
HAVING correlation > 0.85  -- only strongly correlated pairs are usable as test/control
ORDER BY correlation DESC

Power Analysis Before You Commit Budget to a Test

Before launching any incrementality test, run a power analysis: given your typical outcome variance and expected effect size, how many geos (or how long a test duration) do you need to detect a real lift at acceptable confidence? Skipping this step is how teams end up running an inconclusive test for a month, pausing meaningful ad spend in control geos, and learning nothing because the test was underpowered to detect the effect size that was actually plausible.

User-Level Randomization: PSA/Ghost Ads

For channels that support user-level randomization (most programmatic and social platforms), a cleaner design than geo holdouts is a Ghost Ads or PSA (public service announcement) test: randomly assign individual users to either see your actual ad or a placebo/control ad (a PSA, or a 'ghost' impression that's logged but not actually shown), then compare conversion rates between groups. This avoids the geo-comparability problem entirely since randomization happens at the user level — the tradeoff is it requires platform support (Meta's Conversion Lift, Google's Conversion Lift) and sufficient scale to detect effects given typically small per-user conversion rates.

Common Design Flaws That Invalidate Results

- Contamination: control group users seeing the campaign anyway through a channel not controlled by the test (e.g., suppressing paid social but organic/earned exposure to the same message still reaches control users).
- Spillover between geos: treatment geo advertising bleeds into adjacent control geos via shared media markets (a classic issue with TV/radio DMAs that don't align cleanly with desired test boundaries).
- Test duration too short relative to the funnel: pausing a campaign for two weeks when your typical purchase cycle is six weeks measures an incomplete picture of the true effect, understating lift that materializes after the test window closes.
- Ignoring novelty/seasonal confounds: running treatment during a period with an unrelated external event affecting only treatment geos (a regional promotion, a local news event).

Incrementality Tests Have a Real Opportunity Cost

Pausing or suppressing marketing in control geos/segments means genuinely forgoing conversions there for the test's duration — this isn't a free measurement exercise. Size tests to the minimum duration and scope that still achieves adequate statistical power, and be deliberate about which channels and markets justify that cost versus relying on MMM or data-driven attribution as a lower-cost ongoing signal.

What's Next?

Holdout groups are the mechanism underlying most incrementality tests — the next lesson looks specifically at how to structure and maintain a holdout group program over time, beyond a single one-off test.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →