Skip to content
Skip to content
PAID SEARCH AND MEASUREMENT

Microsoft built the right test, then attached the wrong number

Microsoft has handed Performance Max advertisers a holdout test, which is the right instrument for the oldest argument in paid media. It has also attached an 8% uplift figure whose footnote names a period and a geography but never a method, and that gap is exactly what incrementality testing exists to close.

Summarise with
Set as preferred source on Google
19 min read
Quick answerIncrementality testing holds part of your audience back from seeing ads, then compares them with the treated group. The gap is the campaign's true contribution. Microsoft's new Performance Max uplift test runs that comparison inside the platform.
SECTION 01

What incrementality testing actually measures

incrementality testing

Every advertiser has the same argument in the same meeting. Marketing says the channel is working, finance says those customers were coming anyway, and the dashboard sits between them looking confident about a question it was never built to answer. The question is the counterfactual: what would have happened if the campaign had never run at all. Incrementality testing is the only method in the room that answers it, because it deliberately keeps a slice of the audience in the dark and watches what they do without you.

Attribution allocates credit for conversions that already happened. Incrementality testing manufactures a comparison that would not otherwise exist: a treated group that sees the advertising, and a holdout group that does not. The difference between the two is the campaign's contribution, counted carefully rather than assumed.

The distinction bites hardest on brand terms. A branded click looks magnificent in a last-click report, because the buyer typed your name, walked to your door and knocked. Paying for the knock is not the same as paying for the visit, and only a holdout separates them. Follow the scent, not the flattering figures.

It helps to name what the method is not. It is not a media mix model, which infers contribution from history rather than building a comparison. It is not a lift study bolted on after the fact, and it is not something you can back into by setting two campaigns side by side that never shared an audience. The mechanism is one stubborn thing: somebody was withheld, deliberately, and their behaviour became the baseline. Every other approach to measurement is inference.

This one is observation, which is why it costs you conversions in the short run and buys you certainty in the long one. Name that trade out loud before anybody signs the test off, because somebody will notice the missing conversions in week two and it should not be a surprise.

The counterfactual is the only comparison finance finds falsifiable

Finance directors are not hostile to marketing, they are hostile to numbers that cannot be wrong. A before-and-after chart can never be wrong, which is exactly its weakness: seasonality, stock levels, pricing moves and a rival's outage all hide inside the same hopeful line. A holdout can be wrong, publicly and quickly. Incrementality testing hands the finance floor a falsifiable claim, and falsifiable claims are the ones that survive the budget conversation.

Two warnings before the mechanics. The method measures the campaign you actually ran, in the market you actually ran it in, during the weeks you ran it: a local answer, not a law of nature. And it measures the margin, not the whole. A campaign can be genuinely incremental and still be a poor purchase, which is a pricing problem rather than a measurement one.

SECTION 02

The evidence has been lying in the hedgerow for twelve years

None of this is new, which is the quietly embarrassing part. In 2014, Blake, Nosko and Tadelis published a large-scale field experiment on eBay's paid search spending through the National Bureau of Economic Research as working paper 20171. It reached Econometrica the following year, volume 83, pages 155 to 174, under DOI 10.3982/ECTA12423.

The findings are blunt. The authors report in working paper 20171 that returns from paid search are a fraction of conventional non-experimental estimates, and that brand-keyword ads have no measurable short-term benefits. Read the second finding twice, then look at the line item nobody in your account wants to touch.

The follow-up work is, if anything, harsher. Gordon, Zettelmeyer, Bhargava and Chapsky put observational methods on trial in Marketing Science in 2019, using 15 United States advertising experiments at Facebook covering 500 million user-experiment observations and 1.6 billion ad impressions. The observational models often failed to reproduce the effects the randomised experiments produced, even after conditioning on extensive demographic and behavioural variables, and modern causal inference methods did not rescue them.

Two papers, two platforms, one conclusion. The data you already hold cannot tell you what your advertising caused. Only a group you withheld it from can.

Twelve years. That is how long the profession has held rigorous experimental evidence that last-click flatters paid search, and how long the same profession has kept buying on last click anyway. The trail was trodden, marked and published in a journal, and the industry strolled straight past it.

Twelve years of experimental evidence beats one vendor's headline, in every account, every quarter.
folkfox, on why incrementality testing outranks a case study

The study still matters, and not because eBay is your business. The gap it found between experimental and non-experimental estimates is the same gap sitting quietly in your dashboard, and incrementality testing is the instrument that measures the size of it. Platforms change, campaign types change, creative changes. The counterfactual does not.

There is a second, subtler lesson buried in that paper. The bigger and better known the brand, the more its paid search looks like a toll on traffic it already owned. Small challengers often find genuine uplift where incumbents find expensive duplication, which is why folkfox treats the same test as a different decision depending on where a brand sits in its market. That thinking runs through our PPC work, our paid social services and the positioning questions we handle inside brand strategy.

SECTION 03

What Microsoft shipped, and the number we had to go back for

Microsoft has added two Performance Max experiment types, according to Search Engine Journal: uplift tests, which measure what Performance Max adds alongside your existing campaigns, and upgrade tests, which compare a migrated campaign with the one it replaced. The first is a proper holdout design. The second is a switching study, and the two answer different questions.

The two Performance Max experiment types described by <a class="ff-src" href="https://www.searchenginejournal.com/microsoft-advertising-adds-ai-visibility-insights-pmax-testing-and-creative-preview-updates/584760/" target="_blank" rel="noopener noreferrer">Search Engine Journal</a>, and the question each one can honestly answer.
Test typeWhat it comparesThe question it answers
Uplift testPerformance Max running alongside existing campaigns, against a holdoutWhat does this campaign add that we would not have had anyway
Upgrade testA migrated campaign against the campaign type it replacedDid the swap help, hurt or change nothing at all
NeitherThis quarter against last quarterAlmost nothing, because nothing was held back

The same report cites an average 8% increase in incremental conversions, and that figure sent folkfox looking for its home. The first sweep, on 10 August 2026, came back empty. No 8% in the June 2026 product news post, none on the Performance Max solutions page, which to this day carries no uplift figure at all, and nothing in the blog index between June and August 2026.

A second pass found it, and the correction belongs in the body rather than buried at the bottom. The figure is Microsoft's own. The Microsoft Advertising blog of 7 May 2026 states that advertisers using Performance Max see an average 8% increase in incremental conversions, and footnotes it as Microsoft internal data, global, September 2024 to September 2025. It is repeated in the Activate 2026 takeaways of 19 June 2026, alongside an experiments framework and new customer acquisition goals framed around incrementality. A sweep that misses the right page is a sweep that was wrong, and saying so costs less than defending it.

The correction changes one thing and confirms another. It changes the attribution: the 8% is first-party rather than a trade-press invention, and folkfox looked in three places that did not have it. What it confirms is the shape of the evidence. Twelve months and a global footprint tell you when and where. They do not tell you the test design, the sample, the size of the holdout, or what Microsoft counts as an incremental conversion. A period and a place are provenance. Provenance is not a method.

The operating numbers stay with the trade report. The suggested threshold of at least 30 conversions in the previous 30 days and the four to twelve week evaluation window come from Search Engine Journal, and folkfox has not found either of them stated on a Microsoft page.

None of that is an accusation, it is a category note, and it survives the correction intact. A first-party figure with a dated footnote is a claim with a paper trail. A published method, with a sample, a split and a definition of the metric, is evidence. Treating the first as the second is exactly the habit incrementality testing exists to break.

Compare it with the numbers Microsoft does publish

The Microsoft Advertising blog from November 2024 claims a three to four times higher conversion rate on average for small and medium businesses, a 32% decrease in cost per acquisition, and a three times increase in return on ad spend when Performance Max runs alongside search. Those claims carry a footnote naming their basis: Microsoft internal data, small and medium business accounts that adopted Performance Max exclusively or opened new Performance Max accounts, October 2024.

Give Microsoft genuine credit for both footnotes, because plenty of platforms publish nothing of the kind. Then hold the line anyway. Two dated notes on two different pages, each pointing at internal data nobody outside Redmond can rerun, still leave you buying on somebody else's average. Your own uplift test is the reply, and it costs less than the argument it ends.

Microsoft already publishes the right definition, elsewhere

The quiet irony is that the company documents the method properly on a different site. The Microsoft Learn documentation for Xandr, updated in October 2025, defines incrementality as the percentage of conversions attributable to a campaign, states plainly that last-touch and multi-touch attribution do not prove causality, and describes a data feed built on an audience randomly split into a test group exposed to ads and a control group that is not. That page is correct, current and almost never quoted in a pitch.

SECTION 04

Designing incrementality testing that survives a finance meeting

A good test is boring by design. You decide the question before you see the data, you write the decision rule down, and you agree in advance what result would make you stop spending. Everything after that is patience. The prowl matters more than the pounce.

The shape of a defensible test
Fix the question

Write one sentence: what would have happened without this campaign. Name the conversion, the market and the window before anything is switched on.

Build the holdout

Withhold a random, meaningful share of the audience or geography. Random beats convenient, because a convenient holdout is usually a different kind of customer.

Set the decision rule

Agree the uplift that keeps the budget, the uplift that grows it, and the result that kills it. Signed off before launch, by the person who owns the money.

Leave it alone

No mid-flight bid tinkering, no fresh creative, no budget rescue in week three. Every change contaminates the comparison and hands you an unusable result.

Read the gap, not the graph

Compare treated against held-back, state the range rather than a single figure, and record what you would need to see to change your mind next quarter.

On timing and volume, the guidance reported by Search Engine Journal is at least 30 conversions in the previous 30 days and an evaluation window of four to twelve weeks. Treat those as a floor rather than a target. Thirty conversions is enough to start a test, not enough to settle a disagreement, and an account running the experiment across a fortnight of thin data is measuring noise with great precision.

What a holdout actually looks like
Line chart showing indexed conversions for a treated group rising above a flatter holdout group across eight weeks1289664320W1W2W3W4W5W6W7W8Treated group: 100Treated group: 104Treated group: 109Treated group: 113Treated group: 118Treated group: 121Treated group: 125Treated group: 128Holdout group: 100Holdout group: 101Holdout group: 103Holdout group: 102Holdout group: 105Holdout group: 106Holdout group: 108Holdout group: 109
Treated groupHoldout group
Illustrative shape of an eight-week uplift test: the treated group climbs, the holdout drifts, and the gap between them is the only number that matters. Illustrative figures, modelled to explain the method, not measured in any account.

How the same test reads at Google

Google documents its version in the open, which makes it a useful comparator rather than a rival. Google Ads Help calls Conversion Lift an incrementality tool, splits the audience into a treatment group that sees the ads and a control group that does not, and gates access behind an account representative rather than a self-serve switch.

It also publishes the trade-off most vendors leave out. Google's comparison of lift types sets a user-based split, which reports results broken out by conversion action, age, gender and country but suffers measurement gaps, against a geographic split, which is noisier and cannot be sliced by demographics yet covers online and offline conversions across channels and typically needs a bigger budget.

Put those pages beside the 8% and the asymmetry is plain. One vendor publishes the design and its compromises. The other publishes the result. Both may be perfectly honest, and only one of them is checkable, which is the whole reason to run the test yourself.

Three ways these tests quietly fail

The first failure is contamination. Your holdout sees the campaign anyway, through a shared device, a second market or a retargeting pool that nobody remembered to exclude, and the gap collapses to nothing. The second is power: too few conversions, too short a window, and a result that could have come from a slow Tuesday. The third is the goalpost shuffle, where a disappointing outcome becomes a pilot, then a learning, then a footnote.

Run the same discipline across channels and the results start compounding. A holdout is as buildable in paid social as it is in search, and the same logic applies to install campaigns inside app marketing, where organic uplift and paid uplift are notoriously easy to confuse. Google's own version of the argument, which we covered in campaign mix experiments, points the same way.

SECTION 05

Where the demand sits, and what snags advertisers on the way

A short detour into first-party data, because it is a small lesson in mispriced attention. Measuring United States search demand on 10 August 2026 via DataForSEO, folkfox found "incrementality testing" running at 480 monthly searches with a keyword difficulty of 2 and a cost per click of $132.53, trending up 14.8%. A difficulty of 2 means almost nobody is competing for the term organically. A cost per click of $132.53 means the people typing it are worth a fortune.

The term nobody is writing for

Monthly searches

480

United States volume for "incrementality testing", measured by folkfox on 10 August 2026.

Keyword difficulty

2

Out of 100. The organic hedgerow is effectively empty.

Cost per click

$0.00

What advertisers pay for one click on the same intent.

That combination is the whole argument in miniature. Expensive intent with cheap organic competition is the clearest signal there is that a market cares about a subject its suppliers refuse to explain plainly. The quarry is standing in the open, and the field is empty.

Keyword difficulty, same five terms
microsoft advertising
32
performance max
22
microsoft performance max
13
incrementality testing
2
ppc testing
0
Difficulty scores measured by folkfox via DataForSEO on 10 August 2026. The measurement vocabulary is wide open, the brand term is not.

Back to the account, and to the practical snag that catches regulated advertisers before any experiment begins. Search Engine Land reports that Microsoft has rolled out bulk editing and bulk appeals for disapproved text and image assets to all accounts, with advertisers able to submit up to 5,000 ad exception requests. Existing images cannot be edited in place and have to be replaced by upload, every disapproved asset in an ad must be appealed together, and approved assets keep serving during review as long as the final URL stays approved.

Read that as a measurement problem, because that is what it becomes. A disapproval that silently thins your asset pool mid-test is contamination wearing a compliance badge. Any brand in a licensed category should sweep disapprovals before the window opens, which is the same discipline behind our notes on affiliate licence verification and on limited ad serving.

The Microsoft Clarity release belongs to the same product bundle and the same instinct: show advertisers where they are absent rather than only where they are present. Absence is the hardest thing to see in any reporting system, which is precisely why incrementality testing works. It builds an absence on purpose, then measures it. If you want that thinking applied to visibility rather than spend, our companion piece on AI visibility measurement instability covers the same failure mode, and the content side of it lives in content marketing and SEO and GEO services.

Questions

Frequently asked questions

What is incrementality testing in paid advertising?

It is a controlled experiment that withholds advertising from a random share of your audience and compares them with the group that saw it. The difference is the campaign's incremental contribution, which is the number attribution reports cannot produce because they only see conversions that already happened.

How is an uplift test different from an upgrade test in Performance Max?

An uplift test measures what Performance Max adds alongside your existing campaigns, using a holdout. An upgrade test compares a migrated campaign with the campaign type it replaced. The first answers whether the spend is incremental, the second answers whether the swap was worth making.

Did Microsoft publish the 8% incremental conversions figure itself?

Yes. It appears on the Microsoft Advertising blog of 7 May 2026, footnoted as Microsoft internal data, global, September 2024 to September 2025, and again in the Activate 2026 takeaways of 19 June 2026. It does not appear on the Performance Max solutions page, and no test design or sample size is published with it.

How long should an incrementality test run?

Search Engine Journal reports guidance of a four to twelve week evaluation window with at least 30 conversions in the previous 30 days. Lower-volume accounts should lean towards the longer end, because a short window on thin data produces a confident answer to the wrong question.

Why does last-click attribution overstate paid search returns?

Because it credits the ad that closed the visit rather than the advertising that caused it. Blake, Nosko and Tadelis found experimentally that returns from paid search are a fraction of conventional non-experimental estimates, and that brand-keyword ads showed no measurable short-term benefits.

Can smaller accounts run incrementality testing at all?

Yes, with adjusted expectations. Below roughly 30 conversions a month you can still run a geographic holdout over a longer window, but treat the output as a direction rather than a decimal. An under-powered test is not useless, it is just imprecise, and it should be labelled that way.

Keep reading

Read more on this topic

Stop arguing about attribution, start measuring the gap

folkfox designs incrementality testing that holds up in a finance meeting: a real holdout, a decision rule agreed before launch, and a result nobody has to take on trust.

The den

Where to?

Pricing

Choose a section. Enter opens it, Escape continues reading.

Cookie preferences

folkfox uses data the way we use strategy: only when it earns its place.