Skip to content
Skip to content
SEO AND GEO

A single reading is a rumour , not a measurement

Two things landed in the same week: fresh evidence that identical prompts do not return identical answers, and the first serious attempt to say what actually counts as a measurement. Both point at the same uncomfortable place.

Summarise with
Set as preferred source on Google
16 min read
Quick answerAI visibility tracking measures how often AI engines mention your brand. Google documents that its AI features vary between runs, so one reading is a rumour: cadence, sample size and repeat count decide whether the number is usable.
Section 01

Why a single ai visibility tracking run is a rumour

ai visibility tracking

The fox does not judge a river by one paw print pressed in the mud. It watches the bank across several moonlit nights before it commits to a crossing. That patient prowl is exactly the posture this market skipped, because ai visibility tracking arrived as a product long before it arrived as a discipline.

Start with the platform's own paperwork rather than a vendor's pitch. Google's Search Central documentation on AI features states it flatly: "AI Mode and AI Overviews may use different models and techniques, so the set of responses and links they show will vary." The same page confirms that indexing and serving are never guaranteed, and that AI Overviews often do not trigger at all.

Read that twice, because it settles the argument before the vendors get to have it. If the platform documents that its own output varies, a single ai visibility tracking sweep is one sample of a shifting system, and a sample of one carries no error bars. It is a rumour with a chart drawn round it.

The number moves while you are reading it

Academic work published on 6 August 2026 sharpens the same point. In What Current AI Benchmarks Leave Unmeasured, Encarnación, Behzad, Lurie and Metaxa drew a stratified sample of 401 prompts from two benchmarks and collected 4,812 responses across three repeated runs per prompt. Repeated runs of the same prompt produced inconsistent responses in up to 21% of prompts, and enabling web search reduced accuracy by up to 8 percentage points.

The instability study, in four numbers

Prompts sampled

401

Stratified across two benchmarks, BBQ and SafetyBench.

Responses collected

4812

Three repeated runs per prompt, every run recorded.

Prompts that diverged

0%

Upper bound on inconsistent responses across repeats.

Accuracy lost to search

0pp

Upper bound on the drop when web search was enabled.

Repeating the same prompt three times was enough to shake loose inconsistent answers on up to a fifth of prompts.

Be precise about what that study is, because precision is the whole point of the piece. It compared ChatGPT's chat interface against OpenAI's API, so it is not a cross-vendor sweep, and it measured benchmark accuracy rather than brand citations. Carrying the finding across to brand visibility is folkfox's inference, not a claim the authors make. What it establishes cleanly is that repeated identical prompts diverge, which is the assumption every ai visibility tracking dashboard quietly needs to be false.

Section 02

What the instability evidence supports, and what it does not

One study is a scent, not a trail. The useful test is whether the same shape shows up in an unrelated domain, measured by different people, with a different method. It does.

A May 2026 preprint from Zhou, Choi, Wen and Ye ran 16 models from five provider families over 100 coding problems, using two prompt templates and five repeated runs each, for 16,000 evaluation instances. Run-level pass rate overstated retry-free coverage by up to 17.8 percentage points. The domain is code generation rather than search citations, so treat it as corroboration rather than proof of anything about brands. The transferable finding is blunt: single-run measurement flatters the system it measures.

That is the entire case for cadence in ai visibility tracking. If one run flatters, the fix is not a cleverer run, it is more of them, reported with the spread rather than the mean.

Two further preprints tell you where effort actually pays, and both are unfashionably deflating. The largest controlled citation study to date, 252,000 trials across six models and 18 factors, found that topical relevance and position in the retrieved set dominate, that recency and explicit specifics give consistent gains, and that formatting-only edits measure close to zero. A critical survey of 45 GEO studies reaches the same narrow conclusion: relevance and context position are the only reproducible levers, and no reviewed technique shows a stable cross-platform causal effect on organic discoverability.

The honest reading of that hedgerow is bracing. Much of what is sold as generative optimisation has no reproducible effect, the levers that do work are the ones content marketing has always pulled, and the measurement layer sitting on top is noisier than the thing it claims to measure.

Section 03

Directional or decision-grade: the new ai visibility tracking standard

On 3 August 2026 the IAB released Measuring Visibility in the AI Era, and it is the first document in this thicket that reads like a standard rather than a sales deck. It rests on four pillars: Presence, Prominence, Portrayal and Persuasion.

Each pillar carries its own metric set. Presence covers mention rate, citation rate, share of voice and visibility momentum. Portrayal covers sentiment, framing, hallucination rate and factual inaccuracy rate. Persuasion covers recommendation strength and post-citation click-through rate. Prominence sits between them, asking not whether you appear but where you appear inside the answer.

The split that matters most, though, is not the pillars. The IAB separates directional measurement, which it says supports early signal detection and competitive awareness but is not sufficient for budget allocation or executive strategy decisions, from decision-grade measurement, which has to clear a far higher bar on query volume, sample size, prompt type coverage, testing cadence, reproducibility and platform coverage.

The four IAB pillars, the metrics named under each, and the question each pillar forces you to put to a supplier.
PillarMetrics the IAB namesThe question to ask your supplier
PresenceMention rate, citation rate, share of voice, visibility momentumAcross how many distinct prompts, and how many repeats each?
ProminenceWhere you land inside the answer, not merely whether you landIs position recorded, or only presence?
PortrayalSentiment, framing, hallucination rate, factual inaccuracy rateWho reads the answers, a person or a classifier?
PersuasionRecommendation strength, post-citation click-through rateIs the click joined to your own analytics, or estimated?

Set that standard against the state of the market and the gap is comic. Reporting on the release, PPC Land put the share of brands systematically tracking AI visibility at 16%. That figure comes from PPC Land's reporting rather than from the IAB release itself, and in a piece about measurement discipline the difference is not pedantry, it is the point.

Section 04

The budget figures everyone quotes fail their own test

Here is where the trade press hands you a live specimen. Digiday's aggregation of vendor surveys reports an average AI visibility budget share of 24%, with 82% of marketers allocating something, 43% allocating a fifth or more, and 18% allocating nothing at all.

How marketers say they are funding AI search
Donut chart splitting marketers into those allocating a fifth or more of budget to AI visibility, those allocating a smaller share, and those allocating nothingA fifth or more: 43%Something, under a fifth: 39%Nothing: 18%43%
A fifth or more 43%Something, under a fifth 39%Nothing 18%
Four in ten marketers claim to put a fifth or more of budget behind AI search, though none of the aggregated surveys discloses a sample size or a method.

Those figures will be quoted in board briefings all week. Not one of the aggregated surveys discloses a sample size or a method. That is precisely the directional rather than decision-grade problem the IAB names, running loose in the wild, inside the very numbers being used to justify ai visibility tracking spend.

Use them as weather, never as a map. They tell you the category is being funded, which is genuinely useful; they cannot tell you what to fund, which is what a budget owner needs. A spending line built on an undisclosed sample is a burrow dug in sand.

A number without a method is not a measurement. It is a mood with a decimal place.
folkfox, on reading AI visibility research

The same standard has to survive contact with your own dashboard, which is the harder half. If your board slide cannot state prompt count, repeat count and the dates the sweep ran, it is a mood too, however handsome the trend line looks. Treat that as a brand strategy exposure before you treat it as a reporting chore, because the number will outlive the meeting.

Section 05

Building an ai visibility tracking cadence that survives an audit

Search demand is quietly settling this argument on its own. Measuring United States search demand on 10 August 2026 through DataForSEO, folkfox's own keyword sweep found "ai visibility tools" at 1,600 monthly searches with a difficulty of 16 and a trend of plus 420.2%, "ai visibility tracking" at 390 searches with a difficulty of 21 and a trend of plus 392.3%, "ai visibility checker" at 260 searches with a difficulty of 5 and a trend of plus 277.8%, "ai visibility score" at 140 searches with a difficulty of 17 and a flat trend, and "ai visibility audit" at 110 searches with a difficulty of 6 and a trend of minus 47.4%.

Monthly United States searches, measured 10 August 2026
Monthly United States searches, measured 10 August 2026Bar chart of monthly United States search volume for five AI visibility terms, with tools highest at 1,600 and audit lowest at 110tools: 1600tracking: 390checker: 260score: 140audit: 110160012008004000toolstrackingcheckerscoreaudit
Tooling terms carry the volume, while the one-off audit sits at the bottom of the pile, on folkfox's own DataForSEO measurement of 10 August 2026.
Direction of travel, same five terms
tools
rising 420.2%
tracking
rising 392.3%
checker
rising 277.8%
audit
falling 47.4%
score
flat 0.0%
Bars show the size of the move, not its sign: the first three terms are rising sharply, score is flat and audit is falling, on folkfox's own DataForSEO measurement of 10 August 2026.

Read the two charts together and the story is hard to miss. Appetite for continuous tools and for ai visibility tracking is climbing steeply while the one-off audit is quietly draining away. The market is working out for itself that a snapshot is not a measurement, which is the cheerful part of an otherwise stern piece.

Instrumentation is catching up too, and some of it costs nothing. Microsoft's Clarity now labels each grounding query as branded or non-branded, breaks Share of Authority down by that split and lets you filter the dashboard by it, as Clarity's release notes set out and Search Engine Land confirmed. Two days later Clarity added Topic Insights, grouping AI citations by subject so gaps in topic coverage show up as gaps. No adoption or impact figures were published with either release, so treat both as new instruments rather than proven outcomes.

Kaleigh Moore
Right now, at most companies, "figure out AI search" is being handled by seven different people who don't know they're supposed to be working together...
August 2026View on X

Tidy, quotable, unreproducible

Our AI visibility score rose from 41 to 47 this month, putting us ahead of two competitors.

Duller, defensible, repeatable

Across 120 frozen prompts run three times each on four platforms, weekly through August, mention rate averaged 47% with a run-to-run spread of 9 points. Method and prompt set are versioned and attached.

None of this is exotic, which is the quiet consolation. It is the same measurement hygiene paid search learned a decade ago, applied to a channel where the platform itself warns you that the output moves. Get the cadence right and ai visibility tracking becomes a trend you can defend in front of a finance director; get it wrong and it stays a monthly rumour with a chart around it.

Brands who want this built rather than described usually start with SEO and GEO, then carry the same cadence into FinTech marketing and iGaming marketing, where a mis-stated claim is a compliance matter rather than a metric. It sits alongside our work on being cited but never recommended, on the brands models already know, and on what blocking AI crawlers really costs. Today's companion pieces on incrementality testing and open banking fee rules make the same argument in two very different rooms.

Questions

Frequently asked questions

How often should ai visibility tracking actually run?

More often than most suppliers sell it. Google documents that its AI features vary between runs, and a 2026 arXiv study found repeated identical prompts returned inconsistent responses in up to 21% of cases. Weekly sweeps across a frozen prompt set, with several repeats per prompt, give you a trend you can defend. A quarterly one-off gives you a single sample of a shifting system, which is closer to a rumour than a reading.

Is a one-off AI visibility audit worth paying for?

As a starting map, yes. As a measurement, no. Buy an audit to find your gaps and to build your prompt set, then throw away the headline number. A single sweep carries no spread and no repeat, so nothing in it is reproducible. Demand data backs the instinct: search interest in the one-off audit is falling while interest in continuous tools climbs, on folkfox's own keyword measurement of 10 August 2026.

Why do two tools report different numbers for the same brand?

Because they are not measuring the same thing. Prompt sets differ, repeat counts differ, platform coverage differs, and the underlying engines are documented to vary between runs. Ask both suppliers for prompt count, repeats per prompt, platforms covered and the sweep dates. Where those four answers differ, the numbers were never comparable in the first place, and averaging them makes the confusion worse rather than better.

What makes ai visibility tracking decision-grade rather than directional?

The IAB sets the bar at query volume, sample size, prompt type coverage, testing cadence, reproducibility and platform coverage. Directional data is fine for early signal detection and competitive awareness, and the IAB says plainly that it is not sufficient for budget allocation or executive strategy. The working test is simple: if you cannot reproduce last month's number by rerunning the same prompts, you hold directional data, whatever the invoice claims.

Does Microsoft Clarity replace an ai visibility tracking tool?

Not yet, though it changes what you can see without paying. Clarity now labels each grounding query as branded or non-branded, splits Share of Authority by that division, and groups citations by topic. That tells you which subjects earn citations and whether they arrive on your name or on a category term. It reports your own citations rather than the full competitive picture, so treat it as one instrument in the kit.

What should I optimise for if I want to be cited more often?

Relevance and position, on the current evidence. A controlled study of 252,000 trials found topical relevance and position in the retrieved set dominate, with recency and explicit specifics giving consistent gains and formatting-only edits measuring near zero. A survey of 45 GEO studies reached the same conclusion. Write specific, current, self-contained passages on subjects you genuinely cover, and treat any tactic that promises lift without touching relevance with suspicion.

Keep reading

Read more on this topic

Want a number you can defend?

We build ai visibility tracking with the method attached: a frozen prompt set, repeats on every sweep, and reporting that shows the spread instead of hiding it.

The den

Where to?

Pricing

Choose a section. Enter opens it, Escape continues reading.

Cookie preferences

folkfox uses data the way we use strategy: only when it earns its place.