Microsoft built the right test, then attached the wrong number
Microsoft has handed Performance Max advertisers a holdout test, which is the right instrument for the oldest argument in paid media. It has also attached an 8% uplift figure whose footnote names a period and a geography but never a method, and that gap is exactly what incrementality testing exists to close.
By Katie Delaney · 2026-08-10 · 16 min read
What incrementality testing actually measures

Every advertiser has the same argument in the same meeting. Marketing says the channel is working, finance says those customers were coming anyway, and the dashboard sits between them looking confident about a question it was never built to answer. The question is the counterfactual: what would have happened if the campaign had never run at all. Incrementality testing is the only method in the room that answers it, because it deliberately keeps a slice of the audience in the dark and watches what they do without you.
Attribution allocates credit for conversions that already happened. Incrementality testing manufactures a comparison that would not otherwise exist: a treated group that sees the advertising, and a holdout group that does not. The difference between the two is the campaign's contribution, counted carefully rather than assumed.
The distinction bites hardest on brand terms. A branded click looks magnificent in a last-click report, because the buyer typed your name, walked to your door and knocked. Paying for the knock is not the same as paying for the visit, and only a holdout separates them. Follow the scent, not the flattering figures.
It helps to name what the method is not. It is not a media mix model, which infers contribution from history rather than building a comparison. It is not a lift study bolted on after the fact, and it is not something you can back into by setting two campaigns side by side that never shared an audience. The mechanism is one stubborn thing: somebody was withheld, deliberately, and their behaviour became the baseline. Every other approach to measurement is inference.
This one is observation, which is why it costs you conversions in the short run and buys you certainty in the long one. Name that trade out loud before anybody signs the test off, because somebody will notice the missing conversions in week two and it should not be a surprise.
The counterfactual is the only comparison finance finds falsifiable
Finance directors are not hostile to marketing, they are hostile to numbers that cannot be wrong. A before-and-after chart can never be wrong, which is exactly its weakness: seasonality, stock levels, pricing moves and a rival's outage all hide inside the same hopeful line. A holdout can be wrong, publicly and quickly. Incrementality testing hands the finance floor a falsifiable claim, and falsifiable claims are the ones that survive the budget conversation.
Two warnings before the mechanics. The method measures the campaign you actually ran, in the market you actually ran it in, during the weeks you ran it: a local answer, not a law of nature. And it measures the margin, not the whole. A campaign can be genuinely incremental and still be a poor purchase, which is a pricing problem rather than a measurement one.
The evidence has been lying in the hedgerow for twelve years
None of this is new, which is the quietly embarrassing part. In 2014, Blake, Nosko and Tadelis published a large-scale field experiment on eBay's paid search spending through the National Bureau of Economic Research as working paper 20171. It reached Econometrica the following year, volume 83, pages 155 to 174, under DOI 10.3982/ECTA12423.
The findings are blunt. The authors report in working paper 20171 that returns from paid search are a fraction of conventional non-experimental estimates, and that brand-keyword ads have no measurable short-term benefits. Read the second finding twice, then look at the line item nobody in your account wants to touch.
The follow-up work is, if anything, harsher. Gordon, Zettelmeyer, Bhargava and Chapsky put observational methods on trial in Marketing Science in 2019, using 15 United States advertising experiments at Facebook covering 500 million user-experiment observations and 1.6 billion ad impressions. The observational models often failed to reproduce the effects the randomised experiments produced, even after conditioning on extensive demographic and behavioural variables, and modern causal inference methods did not rescue them.
Two papers, two platforms, one conclusion. The data you already hold cannot tell you what your advertising caused. Only a group you withheld it from can.
Twelve years. That is how long the profession has held rigorous experimental evidence that last-click flatters paid search, and how long the same profession has kept buying on last click anyway. The trail was trodden, marked and published in a journal, and the industry strolled straight past it.
Twelve years of experimental evidence beats one vendor's headline, in every account, every quarter.
The study still matters, and not because eBay is your business. The gap it found between experimental and non-experimental estimates is the same gap sitting quietly in your dashboard, and incrementality testing is the instrument that measures the size of it. Platforms change, campaign types change, creative changes. The counterfactual does not.
There is a second, subtler lesson buried in that paper. The bigger and better known the brand, the more its paid search looks like a toll on traffic it already owned. Small challengers often find genuine uplift where incumbents find expensive duplication, which is why folkfox treats the same test as a different decision depending on where a brand sits in its market. That thinking runs through our PPC work, our paid social services and the positioning questions we handle inside brand strategy.
What Microsoft shipped, and the number we had to go back for
Microsoft has added two Performance Max experiment types, according to Search Engine Journal: uplift tests, which measure what Performance Max adds alongside your existing campaigns, and upgrade tests, which compare a migrated campaign with the one it replaced. The first is a proper holdout design. The second is a switching study, and the two answer different questions.
| Test type | What it compares | The question it answers |
|---|---|---|
| Uplift test | Performance Max running alongside existing campaigns, against a holdout | What does this campaign add that we would not have had anyway |
| Upgrade test | A migrated campaign against the campaign type it replaced | Did the swap help, hurt or change nothing at all |
| Neither | This quarter against last quarter | Almost nothing, because nothing was held back |
The same report cites an average 8% increase in incremental conversions, and that figure sent folkfox looking for its home. The first sweep, on 10 August 2026, came back empty. No 8% in the June 2026 product news post, none on the Performance Max solutions page, which to this day carries no uplift figure at all, and nothing in the blog index between June and August 2026.
A second pass found it, and the correction belongs in the body rather than buried at the bottom. The figure is Microsoft's own. The Microsoft Advertising blog of 7 May 2026 states that advertisers using Performance Max see an average 8% increase in incremental conversions, and footnotes it as Microsoft internal data, global, September 2024 to September 2025. It is repeated in the Activate 2026 takeaways of 19 June 2026, alongside an experiments framework and new customer acquisition goals framed around incrementality. A sweep that misses the right page is a sweep that was wrong, and saying so costs less than defending it.
The correction changes one thing and confirms another. It changes the attribution: the 8% is first-party rather than a trade-press invention, and folkfox looked in three places that did not have it. What it confirms is the shape of the evidence. Twelve months and a global footprint tell you when and where. They do not tell you the test design, the sample, the size of the holdout, or what Microsoft counts as an incremental conversion. A period and a place are provenance. Provenance is not a method.
The operating numbers stay with the trade report. The suggested threshold of at least 30 conversions in the previous 30 days and the four to twelve week evaluation window come from Search Engine Journal, and folkfox has not found either of them stated on a Microsoft page.
None of that is an accusation, it is a category note, and it survives the correction intact. A first-party figure with a dated footnote is a claim with a paper trail. A published method, with a sample, a split and a definition of the metric, is evidence. Treating the first as the second is exactly the habit incrementality testing exists to break.
Compare it with the numbers Microsoft does publish
The Microsoft Advertising blog from November 2024 claims a three to four times higher conversion rate on average for small and medium businesses, a 32% decrease in cost per acquisition, and a three times increase in return on ad spend when Performance Max runs alongside search. Those claims carry a footnote naming their basis: Microsoft internal data, small and medium business accounts that adopted Performance Max exclusively or opened new Performance Max accounts, October 2024.
Give Microsoft genuine credit for both footnotes, because plenty of platforms publish nothing of the kind. Then hold the line anyway. Two dated notes on two different pages, each pointing at internal data nobody outside Redmond can rerun, still leave you buying on somebody else's average. Your own uplift test is the reply, and it costs less than the argument it ends.
Microsoft already publishes the right definition, elsewhere
The quiet irony is that the company documents the method properly on a different site. The Microsoft Learn documentation for Xandr, updated in October 2025, defines incrementality as the percentage of conversions attributable to a campaign, states plainly that last-touch and multi-touch attribution do not prove causality, and describes a data feed built on an audience randomly split into a test group exposed to ads and a control group that is not. That page is correct, current and almost never quoted in a pitch.
Designing incrementality testing that survives a finance meeting
A good test is boring by design. You decide the question before you see the data, you write the decision rule down, and you agree in advance what result would make you stop spending. Everything after that is patience. The prowl matters more than the pounce.
Write one sentence: what would have happened without this campaign. Name the conversion, the market and the window before anything is switched on.
Withhold a random, meaningful share of the audience or geography. Random beats convenient, because a convenient holdout is usually a different kind of customer.
Agree the uplift that keeps the budget, the uplift that grows it, and the result that kills it. Signed off before launch, by the person who owns the money.
No mid-flight bid tinkering, no fresh creative, no budget rescue in week three. Every change contaminates the comparison and hands you an unusable result.
Compare treated against held-back, state the range rather than a single figure, and record what you would need to see to change your mind next quarter.
On timing and volume, the guidance reported by Search Engine Journal is at least 30 conversions in the previous 30 days and an evaluation window of four to twelve weeks. Treat those as a floor rather than a target. Thirty conversions is enough to start a test, not enough to settle a disagreement, and an account running the experiment across a fortnight of thin data is measuring noise with great precision.
How the same test reads at Google
Google documents its version in the open, which makes it a useful comparator rather than a rival. Google Ads Help calls Conversion Lift an incrementality tool, splits the audience into a treatment group that sees the ads and a control group that does not, and gates access behind an account representative rather than a self-serve switch.
It also publishes the trade-off most vendors leave out. Google's comparison of lift types sets a user-based split, which reports results broken out by conversion action, age, gender and country but suffers measurement gaps, against a geographic split, which is noisier and cannot be sliced by demographics yet covers online and offline conversions across channels and typically needs a bigger budget.
Put those pages beside the 8% and the asymmetry is plain. One vendor publishes the design and its compromises. The other publishes the result. Both may be perfectly honest, and only one of them is checkable, which is the whole reason to run the test yourself.
Three ways these tests quietly fail
The first failure is contamination. Your holdout sees the campaign anyway, through a shared device, a second market or a retargeting pool that nobody remembered to exclude, and the gap collapses to nothing. The second is power: too few conversions, too short a window, and a result that could have come from a slow Tuesday. The third is the goalpost shuffle, where a disappointing outcome becomes a pilot, then a learning, then a footnote.
Run the same discipline across channels and the results start compounding. A holdout is as buildable in paid social as it is in search, and the same logic applies to install campaigns inside app marketing, where organic uplift and paid uplift are notoriously easy to confuse. Google's own version of the argument, which we covered in campaign mix experiments, points the same way.
Where the demand sits, and what snags advertisers on the way
A short detour into first-party data, because it is a small lesson in mispriced attention. Measuring United States search demand on 10 August 2026 via DataForSEO, folkfox found "incrementality testing" running at 480 monthly searches with a keyword difficulty of 2 and a cost per click of $132.53, trending up 14.8%. A difficulty of 2 means almost nobody is competing for the term organically. A cost per click of $132.53 means the people typing it are worth a fortune.
Monthly searches
United States volume for "incrementality testing", measured by folkfox on 10 August 2026.
Keyword difficulty
Out of 100. The organic hedgerow is effectively empty.
Cost per click
What advertisers pay for one click on the same intent.
That combination is the whole argument in miniature. Expensive intent with cheap organic competition is the clearest signal there is that a market cares about a subject its suppliers refuse to explain plainly. The quarry is standing in the open, and the field is empty.
Back to the account, and to the practical snag that catches regulated advertisers before any experiment begins. Search Engine Land reports that Microsoft has rolled out bulk editing and bulk appeals for disapproved text and image assets to all accounts, with advertisers able to submit up to 5,000 ad exception requests. Existing images cannot be edited in place and have to be replaced by upload, every disapproved asset in an ad must be appealed together, and approved assets keep serving during review as long as the final URL stays approved.
Read that as a measurement problem, because that is what it becomes. A disapproval that silently thins your asset pool mid-test is contamination wearing a compliance badge. Any brand in a licensed category should sweep disapprovals before the window opens, which is the same discipline behind our notes on affiliate licence verification and on limited ad serving.
The Microsoft Clarity release belongs to the same product bundle and the same instinct: show advertisers where they are absent rather than only where they are present. Absence is the hardest thing to see in any reporting system, which is precisely why incrementality testing works. It builds an absence on purpose, then measures it. If you want that thinking applied to visibility rather than spend, our companion piece on AI visibility measurement instability covers the same failure mode, and the content side of it lives in content marketing and SEO and GEO services.
Frequently asked questions
What is incrementality testing in paid advertising?
It is a controlled experiment that withholds advertising from a random share of your audience and compares them with the group that saw it. The difference is the campaign's incremental contribution, which is the number attribution reports cannot produce because they only see conversions that already happened.
How is an uplift test different from an upgrade test in Performance Max?
An uplift test measures what Performance Max adds alongside your existing campaigns, using a holdout. An upgrade test compares a migrated campaign with the campaign type it replaced. The first answers whether the spend is incremental, the second answers whether the swap was worth making.
Did Microsoft publish the 8% incremental conversions figure itself?
Yes. It appears on the Microsoft Advertising blog of 7 May 2026, footnoted as Microsoft internal data, global, September 2024 to September 2025, and again in the Activate 2026 takeaways of 19 June 2026. It does not appear on the Performance Max solutions page, and no test design or sample size is published with it.
How long should an incrementality test run?
Search Engine Journal reports guidance of a four to twelve week evaluation window with at least 30 conversions in the previous 30 days. Lower-volume accounts should lean towards the longer end, because a short window on thin data produces a confident answer to the wrong question.
Why does last-click attribution overstate paid search returns?
Because it credits the ad that closed the visit rather than the advertising that caused it. Blake, Nosko and Tadelis found experimentally that returns from paid search are a fraction of conventional non-experimental estimates, and that brand-keyword ads showed no measurable short-term benefits.
Can smaller accounts run incrementality testing at all?
Yes, with adjusted expectations. Below roughly 30 conversions a month you can still run a geographic holdout over a longer window, but treat the output as a direction rather than a decimal. An under-powered test is not useless, it is just imprecise, and it should be labelled that way.
Google's Campaign Mix Experiments
Google's answer to the same question, and what its experiment design does and does not hold constant.
Read the pieceLinkedIn publishes four numbers for one ad format. None of them agree
What happens when a platform benchmark is quoted four different ways in the same week.
Read the pieceWho verifies the verifier, now the arbiter has an owner
Ad verification consolidates, and the independence that made the numbers useful gets harder to prove.
Read the pieceGoogle Ads AI Max arrives on 1 September whether you asked or not
An automatic upgrade is a change to your account that nobody held back a control group for.
Read the piece
Stop arguing about attribution, start measuring the gap
folkfox designs incrementality testing that holds up in a finance meeting: a real holdout, a decision rule agreed before launch, and a result nobody has to take on trust.

