Effectiveness asks: did this marketing activity cause an outcome that wouldn't have happened otherwise? Efficiency asks: how cheaply did we produce the outcomes we got? A campaign can be highly efficient (low cost per conversion) while being almost entirely ineffective (nearly all of those conversions would have happened anyway, with or without the campaign). Confusing the two is the single most common and most expensive measurement mistake in marketing.

The clearest example: retargeting ads often show spectacular efficiency — low CPA, high ROAS — because they're shown almost exclusively to people who already intended to buy. The campaign is efficient at claiming credit for conversions; it may be nearly worthless at causing new ones.

Attribution measures correlation; effectiveness measures causation

An attribution model, however sophisticated, tells you which touchpoints were present before a conversion. It cannot tell you whether the conversion would have happened without that touchpoint. Only a method that compares exposed vs. unexposed groups — a true experiment — can answer the causal question. This is why effectiveness measurement and attribution are complementary, not interchangeable.

Incrementality: the core concept

Incremental lift is the additional outcome caused by a marketing activity, above and beyond what would have happened anyway (the baseline, or "counterfactual"). The only reliable way to estimate a counterfactual is to construct a group that didn't see the activity and compare it to a group that did, holding everything else equal.

Incremental Lift = Outcome(exposed group) − Outcome(unexposed/control group)

If 8% of an exposed group converts and 6.5% of a matched control group converts anyway, the incremental lift is 1.5 percentage points — meaning roughly 81% of the "conversions" in the exposed group would have happened regardless of the campaign (6.5/8), and only about 19% are truly incremental.

Holdout groups: the standard method

A holdout group (also called a control group or ghost ad in some platforms) is a randomly selected segment deliberately excluded from a campaign, so its natural behavior can be compared against the exposed segment. Three practical requirements make a holdout valid:

1. Random assignment — the holdout must be chosen randomly from the eligible audience, not by convenience (e.g., "people we didn't have budget to reach" is not a random holdout and will bias the result).
2. Sufficient size — small holdouts produce noisy lift estimates; the required size grows as the expected lift shrinks, since you're trying to detect a small difference between two numbers.
3. Clean separation — the holdout group must not be inadvertently exposed through another channel (e.g., organic search or a different campaign) or the comparison is contaminated.

holdout_test_results.txt

text

Email win-back campaign — 90-day holdout test
Audience: 200,000 lapsed customers, randomly split 90/10

Group            Size      Reactivated   Reactivation Rate
--------------------------------------------------------------
Exposed (sent)   180,000       10,800           6.00%
Holdout (not sent) 20,000         900           4.50%
--------------------------------------------------------------
Incremental lift:                                1.50 pts
Incremental reactivations: 180,000 x 1.50% = 2,700 customers
Of the 10,800 "attributed" reactivations, only 2,700 (25%)
were actually caused by the campaign.

Effectiveness measurement methods

Method	How it works	Best for	Limitation
Randomized holdout	Withhold a random subset from exposure, compare outcomes	Any channel where you control who gets exposed (email, direct mail, paid ads with ghost ads)	Requires enough volume to detect a statistically reliable lift
Geo experiments	Turn a channel on/off or vary spend across matched geographic markets	Channels where individual-level holdouts aren't possible (TV, OOH, broad paid search)	Geos must be well-matched; slower to read results
Matched market test	Pair similar markets, treat one, leave the other as control	Regional rollouts, new-channel pilots	Finding truly comparable markets is hard
Pre/post comparison	Compare performance before and after a change	Quick directional reads when nothing else is feasible	Confounds any other change happening at the same time — weakest method

Don't let platform-reported lift replace your own test

Walled-garden ad platforms (Meta, Google) offer their own built-in lift studies, but they measure lift only within their own platform's exposure, using their own methodology, and have an obvious incentive to show their channel in the best light. Treat platform-reported lift as directional, and run independent holdout tests for any channel receiving a large share of budget.

Using effectiveness to correct efficiency metrics

Once you have a credible incrementality rate for a channel (say, 25% of attributed conversions are truly incremental, as in the example above), you can build a corrected efficiency metric: True CPA = Spend ÷ Incremental Conversions, rather than spend ÷ all attributed conversions. This number is almost always worse than the platform-reported CPA — sometimes by 3-5x — and it's the number that should actually drive budget allocation decisions, not the flattering platform number.

What's next

Effectiveness tells you whether a channel causes results at all; ROI measurement tells you whether those caused results are worth what you paid for them. The next lesson covers the formulas and common traps in ROI calculation.

Next: Measuring Marketing ROI →

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →