Incrementality Testing in Marketing: How to Measure What Actually Worked
Attribution tells you a story about what happened. Incrementality testing tells you what would have happened if you'd done something different. Why it's the gold standard and how to do it.
Incrementality Testing in Marketing: How to Measure What Actually Worked
Incrementality testing answers the question that attribution can't: did the marketing activity cause conversions that wouldn't have happened otherwise? It's the difference between "we can see that people who saw our ad converted" and "the ad caused people to convert who wouldn't have." The first is correlation. The second is causation. Only the second tells you whether the marketing investment was worth it.
How incrementality testing works
The core design is simple: split your audience into two groups. One group sees the marketing activity (test group). One group doesn't (control group). Measure the difference in conversions between the two groups. The difference is the incremental effect — the conversions caused by the marketing.
The execution is harder than the design because the platforms don't make it easy. Most ad platforms optimize delivery toward people likely to convert, which means the test and control groups are different in ways that confound the results. Proper incrementality testing requires randomization — the two groups must be comparable in every way except whether they were exposed to the marketing.
Types of incrementality tests
Geographic holdout. Run the campaign in some geographic markets and hold it out of others. Compare conversion rates between the test and control markets. Works well for brand advertising and awareness campaigns where the effect should be visible at the market level.
Audience holdout. Within your target audience, randomly withhold a small percentage (typically 10-20%) from seeing the ads. Compare conversion rates between the exposed and held-out groups. Works for direct-response advertising where you can track individual-level conversion behavior.
Platform conversion lift studies. Meta, Google, and other platforms offer built-in lift measurement tools that handle the randomization and measurement. These are easier to execute than DIY holdout tests but less transparent — the platform both runs the test and reports the results.
What incrementality tells you that attribution doesn't
Attribution says: "People who saw your retargeting ad and then converted — the ad gets credit." Incrementality testing says: "People who saw your retargeting ad converted at a rate of 5%. People who didn't see it converted at a rate of 4.8%. The incremental lift was 0.2 percentage points — meaning most of the people who converted after seeing the ad would have converted anyway."
This is the uncomfortable truth that incrementality testing often reveals: a significant portion of attributed conversions aren't incremental. The people would have converted regardless. The ad spend generated attribution credit, not additional revenue. Knowing this allows you to reallocate budget from low-incrementality activities to high-incrementality ones.
When to run incrementality tests
Incrementality testing requires sufficient volume to produce statistically meaningful results — typically thousands of conversions in the test period. It's not practical for small campaigns or low-volume businesses.
Run incrementality tests for major budget decisions: evaluating a new channel before scaling investment, validating that an existing channel is still producing incremental results, comparing the effectiveness of different campaign approaches. Don't run them continuously — the holdout group represents foregone revenue, and the cost of running the test should be justified by the value of the learning.
The incrementality-optimized budget
The implication of incrementality testing: the optimal budget allocation is probably different from what your attribution reports suggest. Channels that look efficient in attribution may have low incrementality. Channels that look expensive in attribution may have high incrementality because they're reaching people who wouldn't convert through other channels.
The brands that do incrementality testing systematically tend to shift budget toward channels that attribution undervalues (especially awareness and upper-funnel activities) and away from channels that attribution overvalues (especially retargeting and branded search). The shift typically improves overall marketing efficiency by 10-30%. The improvement comes from spending less on activities that generate attributed conversions but not incremental ones.