Content Marketing Glossary

  • What is Incrementality?

What is Incrementality?

Incrementality

Incrementality is the measurement of how much marketing activity actually caused a conversion - as opposed to correlated with one. Incrementality testing uses controlled experiments (holdout groups, geo-tests, A/B tests) to compare outcomes between groups that received marketing and groups that didn’t. The difference is the incremental lift. Incrementality is the honest version of attribution because it isolates causation rather than reporting correlation.

Why incrementality matters

Three core reasons:

Attribution models overcredit systematically. A user exposed to retargeting, paid search, and email before converting has all three channels credited in most models. Incrementality reveals which touches actually caused the conversion versus which just happened during the journey.

Natural demand gets misattributed. People who would have converted anyway don’t need the marketing touches they received. Incrementality strips this out.

Bad channels can look good in attribution. A retargeting campaign that shows a 5:1 ROAS in last-click might have a true incremental lift of 1:1 or lower. Incrementality catches this.

The main types of incrementality tests

Four common structures:

Holdout tests. Random sample of users doesn’t receive marketing for a period. Conversion rates between the holdout and control groups reveal the lift.

Geo-tests. Marketing runs in some regions but not others. Comparable regions tested against each other reveal regional lift.

Time-based tests. Spending stopped in specific periods; comparison with active periods reveals cumulative lift.

Ghost-ad tests. Some platforms (Meta, Google) support serving non-ads to control groups - recording what would have been served without actually serving it. Allows ROI comparison without fully suppressing marketing.

What incrementality testing typically reveals

Three patterns recur:

Retargeting is often over-credited. Real incremental lift from retargeting often runs 40–60% of the attribution-reported lift.

Brand search is often over-credited. Users who typed your brand name were going to convert via any available path. Paid search on branded queries often shows lower incremental lift than attribution implies.

Awareness channels are often under-credited. Podcasts, PR, content marketing, video ads often show higher incremental lift than attribution models give them.

These findings shift budget allocation in predictable directions - away from obvious bottom-funnel channels, toward harder-to-attribute top-of-funnel.

Designing a good incrementality test

Six disciplines:

Clear hypothesis. What’s being tested? ‘Does retargeting drive incremental conversions?’ is testable. ‘Does marketing work?’ isn’t.

Control group sizing. Big enough to be statistically meaningful; small enough that the business risk is tolerable.

Duration. Long enough for effects to materialise. Short tests can miss delayed conversions.

Comparable test and control. Groups should be matched on relevant dimensions (demographics, past behaviour, geography). Mismatches contaminate results.

Pre-registered analysis. Define success metrics and analysis approach before seeing results. Prevents post-hoc rationalisation.

Statistical analysis. Proper significance testing, confidence intervals. Incrementality tests produce noisy data; unrigorous analysis produces unreliable conclusions.

When incrementality testing is worth the cost

Four scenarios:

Major channel decisions. Should we keep spending $5M/year on this channel? Worth a proper incrementality test before deciding.

New channel evaluation. New channels often show initial strong attribution results. Incrementality tells you how much is real.

Budget-reallocation decisions. Moving significant budget between channels based on attribution data alone is risky. Incrementality validates the reallocation logic.

Attribution-model validation. Incrementality benchmarks attribution model outputs. Large divergences flag model issues.

When incrementality isn’t worth it

Three cases:

Small channels. Channels representing under 5% of spend often don’t justify the operational complexity.

Fast-changing channels. Channels whose best practices change quarterly may shift before the test completes.

Infrastructure that doesn’t support it. Some marketing contexts don’t permit clean experimentation. Forcing tests anyway produces noise.

Content and incrementality

Three content-specific considerations:

Content incrementality is hard to test. Holding back content from users is hard to implement. Geo-tests can work for paid content distribution, but organic content incrementality is often measured indirectly.

Incrementality reveals content’s real value. When tested properly, content programmes often show higher incremental value than attribution models suggest.

Content supports other channels’ incrementality. Content that pre-warms the audience makes later paid touches more effective. This ‘contribution to other channels’ is hard to isolate but real.

Penfriend’s strategic value often becomes most visible in incrementality-style analysis rather than last-click attribution. Content programmes scaled through Penfriend tend to show disproportionate incremental lift versus their attribution-reported contribution, which is the direction honest measurement usually corrects.

Related terms