Three files tell you the whole strategy

Read the file, not the interview#

There is an old habit worth keeping when a publisher announces a strategy: go and read what it actually serves. USA Today's robots.txt runs to 970 lines and just under sixteen kilobytes, and it names 287 user agents. Of those, 284 are handed a flat disallow across the whole site. That is 99.0 per cent, and it is checkable in one command by anyone who doubts it.

The three exceptions are more instructive than the 284, and they are worth following like a trail through the hedgerow. Googlebot-News is let through with only narrow sponsor-content exclusions. The catch-all asterisk stays permissive. And AmazonAdBot gets an explicit allow. Everything else, the whole scampering pack of them, meets the same shut door.

That 99 per cent figure has been doing the rounds this week as though it were new. It is not. It comes from Mike Reed, Gannett's chairman and chief executive, on the Q3 2025 earnings call, where he said the company was "blocking over 99% of AI-verified bots other than Google that try to scrape our content without licensing agreements in place". That call was ten months ago. What is new is that the file still backs him up, which is the better citation anyway.

The same call carried a sharper number that nobody quoted, and it is the one that should set the scent for the whole piece. In September alone, Reed said, the company blocked 75 million AI bots across its local and national platforms, and roughly 69.9 million of those were OpenAI's. One crawler, more than nine in ten of the blocked traffic. That is not a policy position. That is a siege.

The content licensing paradox in the same file#

Here is where the tidy version of this story falls apart, and where it gets genuinely useful. The reporting says USA Today blocks unapproved crawlers while whitelisting its licensed partners. The file says no such thing.

This matters beyond one newspaper. The traffic every publisher used to live on is thinning, and agency research published this week puts numbers on the thinning: organic sessions across 54 advertisers fell 10.5 per cent over fourteen months, with 46 of the 54 losing ground. When the open trail narrows, the den has to earn its keep another way, and content licensing is the way most publishers have reached for.

Gannett and Perplexity signed a content licensing agreement on 30 July 2025, covering USA TODAY and more than 200 local publications. PerplexityBot is nonetheless given a flat disallow in robots.txt, softened only by a narrow allow on the contributor-content path. GPTBot and OAI-SearchBot get exactly the same treatment. Applebot, Amazonbot, Google-Extended, ClaudeBot and anthropic-ai are simply disallowed outright.

Look closely at what the blocked crawlers are nonetheless allowed, because it is the most revealing line in the file. Both GPTBot and PerplexityBot carry a narrow allow on the contributor-content path while the rest of the site is shut to them. Contributed and sponsored material is opened to the machines; the reported journalism is not. Whatever else that is, it is a valuation, written in a config file, of which words are worth withholding and which are worth giving away for the citation.

So a paying partner is blocked in the public file. The only reading that survives is that partner access is granted somewhere robots.txt cannot see it: at the edge, at the origin, by key, by IP, by contract. Which means the file is not the access-control system at all. It is the notice board outside it.

Meanwhile Bingbot and msnbot are not named anywhere in the file, quietly slipping the fence, so they fall under the permissive asterisk and walk in freely. Microsoft is the network's other licensing partner. Whether that is design or oversight, the effect is the same: what a publisher permits and what its robots.txt says are two different documents, and only one of them is public.

A block is a declaration, not a mechanism#

Robots.txt has never been an enforcement technology, whatever the undergrowth of blog posts insists. The standard itself is a voluntary protocol that describes what a crawler ought to do, and a well-behaved crawler obeys it because it chooses to. The gap between the stated rule and the enforced one is where all the interesting numbers live.

Those figures come from HasData's AI Crawler Block Index, published in July 2026, which tested 10,894 registrable domains, 9,746 drawn from a top-web ranking and 1,148 news publishers, then ran paired requests from identical datacentre addresses, one identifying as GPTBot and one as an ordinary browser, to separate genuine blocking from rate limiting.

Its most useful finding is the uncomfortable one, and it is where most publishers are outfoxed by their own file. On the enforcement subset, 39.5 per cent of GPTBot bans were not actually enforced at all. They existed on paper, in robots.txt, and nowhere else. Two publishers with identical files can therefore have entirely different realities, and only one of them has anything to sell.

The file that returns 200 and isn't there#

If a publisher were genuinely rebuilding itself for machines, the cheapest possible signal would be an llms.txt: a small, well-known convention for pointing a language model at the parts of a site worth reading. Request usatoday.com/llms.txt and you get HTTP 200, which looks like success.

What arrives is 176 kilobytes of the USA Today homepage, served as text/html, titled "Latest World & National News & Headlines". It is a soft 404 wearing a success code. There is no llms.txt. The status line says yes and the body says no, and an automated check that trusts the status line would record a pass.

Pedro Piñera, who builds the Tuist developer platform, found precisely the same failure on his own site three days ago and scored its homepage at 29 out of 100 for citability. A catch-all route swallowed the path and served HTML where markdown should have been. One is a national newspaper and one is a developer tool, and both are serving crawlers a wall while believing they serve a door.

In fairness to USA Today, absence is close to the norm. HasData put llms.txt adoption at 7.4 per cent across the whole sample and just 3.2 per cent among publishers. The point is not that the file is missing. It is that nobody checked, at a company whose stated strategy depends on machines reading it well.

Worth saying plainly: none of this is exotic engineering. Google's own crawling documentation has documented how the file is fetched, cached and interpreted for years, and the checks below take an afternoon. The reason they go undone is that nobody owns them. Search owns rankings, engineering owns uptime, and the machine-readable surface sits in the gap between the two like a burrow nobody has mapped.

Markdown is a test, and the humans are the tell#

The most quoted claim of the week is that USA Today is converting its web pages to markdown. The original report is more careful than its readers have been: the company is testing approaches, one of which is markdown conversion. It has not shipped.

Probing for it finds nothing public. Appending .md to an article URL returns HTML. Sending an Accept header asking for text/markdown returns HTML. That does not refute the claim, since a partner-facing feed would never be exposed to the open web, but it does mean nobody outside the building can verify it, and the article that says the pages "are" markdown is ahead of the evidence.

Two lines from the piece are worth keeping, because together they describe the actual strategy better than the markdown detail does. Reed says the company accepts it must now create and format its content for humans and for machines alike. And Kara Chiles, senior vice president of product management, adds that much of the current work will probably not be apparent to a human reader at all. Both are paraphrased here rather than quoted, because both sentences carry US spellings the house style does not print.

Read those in sequence and the shape emerges. The work is not a redesign. It is the construction of a second, parallel product that only machines consume, negotiated by contract and delivered outside the public file. What does content licensing mean, in that world? It means selling access to a version of your archive that your readers will never see.

The wider trade is moving the same way. Digiday's reporting on the post-traffic era describes an open web that is getting smaller rather than dying, with publishers rebuilding revenue that no longer depends on a session at all. Content licensing is the clearest version of that shift, because it prices the archive instead of the visit.

For everyone else, the practical lesson is smaller and sharper than the headline. Your archive's value to a model company is a function of what it can prove it has, not what you announce. Before the next conversation about content licensing fees, go and read your own three files, because whoever is buying already has. A fox that has not walked its own ground does not get to name the price of it.

Frequently asked questions#