Start free
Kunads / Glossary / Holdout test

Holdout test

Updated 2026-11-19 · Editorial policy

A holdout test measures what advertising adds by keeping a comparable group from seeing the ads and comparing its results with the group that could. The gap between the two, after scaling the holdout to the same size, is the incremental effect: the conversions that would not have happened without the ads.

Incremental conversions = test group conversions − holdout conversions (scaled to the test group's size)

Lift = incremental conversions ÷ scaled holdout conversions

How do Meta and Google run a holdout test?

Both split an audience at random into a group that can see the ads and a group that cannot, then compare conversions. Meta's lift tests follow these steps: choose an objective and campaign to test; divide the intended audience at random into test and holdout groups, which people stay in for the whole test; let the test group have the opportunity to see the ads while the holdout is withheld; measure actions in both groups; and compare them to calculate lift with a confidence percentage. When the groups differ in size, Meta first scales the holdout up to the test group's size.

Google Ads calls its version Conversion Lift and names the groups treatment and control. It offers two kinds of study: one based on users, built from aggregated user attributes, and one based on geography, which also accepts offline data. Depending on the study, Google reports incremental conversions, relative lift, incremental value and incremental ROAS. Google states that Conversion Lift is not available for every account; you may need to go through a Google representative.

Meta adds a warning worth repeating to anyone who reads the result: lift results are specific to the test's conditions and should not be compared with Ads Manager, because lift counts all conversions in both groups while Ads Manager counts conversions attributed to ads within attribution windows.

Worked example: an uneven split

A fashion brand runs a four-week lift test with a 90/10 split.

  1. Groups: test 900,000 accounts, holdout 100,000.
  2. Purchases: test 4,500, holdout 400.
  3. Scale the holdout: the test group is 9 times larger, so 400 × 9 = 3,600 purchases expected without ads.
  4. Incremental purchases: 4,500 − 3,600 = 900.
  5. Lift: 900 ÷ 3,600 = 25%.
  6. Money: average order €70, spend €30,000. Incremental revenue 900 × €70 = €63,000, so incremental ROAS is €63,000 ÷ €30,000 = 2.1×.
Bar chart: the test group made 4,500 purchases, the holdout scaled to the same size would have made 3,600, so the ads added 900 purchases, a 25% lift.

Comparing raw totals (4,500 vs 400) without scaling would be meaningless; the groups are different sizes.

How is a holdout test different from an A/B test?

An A/B test compares two versions of advertising; a holdout test compares advertising with none. Meta draws the line itself: A/B tests do not include a randomised group that is withheld from ads, so they can tell you which creative or audience did better, but not what either adds over showing nothing. If the question is "is this channel worth its budget at all?", only a holdout answers it.

A pause test you can run without a platform tool

A shop can also hold out time instead of people: switch a channel off and see what sales do.

Against a breakeven of 2.0×, Meta as run here costs more than it brings. Against a breakeven of 1.5×, it pays. Pause tests are cruder than randomised ones (anything else that changes in those weeks gets blamed on the pause), but they use your own orders and need no special setup.

A regional holdout, for shops that sell in several areas

Between a randomised lift test and a full pause sits the geographic holdout: switch ads off in some regions and keep them on in comparable others. Suppose a shop sells in six similar regions. It keeps Google Shopping running in three and pauses it in the other three for four weeks. Before the test, both sets of regions produced about 2,000 orders each over four weeks. During the test, the "on" regions produce 2,100 orders and the "off" regions 1,750.

The weak point is the match. If one region had a heatwave or a local competitor closed, the difference is partly that. Pick regions that tracked each other closely before the test, and check that they still did in the weeks before you start.

Where holdout tests go wrong

Holdouts are the measured counterpart to incrementality as a concept, and the check that a marketing mix model should be calibrated against.

See also

Lead generation? Send won deals from your CRM and Kunads ranks campaigns by what the leads were worth, not by how many came in.

Check my lead campaigns Start free for 14 days