Blocking AI crawlers closed one gate. Googlebot was behind it

The 403 that nobody ordered#

A fox does not torch the whole hedgerow to be rid of one rabbit. It watches, waits, works out which gap the quarry actually uses, and closes that single gap. Blocking AI crawlers has become the marketing equivalent of burning the hedgerow flat, and the creature you never meant to catch is usually the one paying the bills.

On 4 August 2026, Search Engine Journal reported a site owner posting in the r/TechSEO community whose sitemap began returning HTTP 403 to Googlebot and bingbot once Cloudflare's AI Training block was set to block. The owner's account was blunt: with the setting enabled the sitemap fetch returns a 403, and disabling it makes the 403 disappear. Google's John Mueller asked the poster to send him a message so he could investigate the case directly.

Be fair to the vendor, because this is a hard problem and publishers begged for the control. Cloudflare shipped a granular version of it. Its AI Crawl Control documentation describes setting allow or block rules for individual crawlers, monitoring which AI services arrive, tracking robots.txt compliance, and even charging for access through Pay Per Crawl. The criticism here is not of the feature. It is of the blunt use of a sharp tool, by people who cannot see what the swing costs.

Bot policing tightens, and blocking AI crawlers is one front of it#

The mood is not confined to CDNs. On the same day, Search Engine Land reported a limited, unconfirmed test in which Google swapped its usual bot check for a prompt asking searchers to sign in to see more results, spotted and shared by Kamlesh Shukla. Google has not confirmed it. Read that as weather rather than forecast: everyone is narrowing the definition of a legitimate visitor, and blocking AI crawlers is the version of that instinct which lands on your own infrastructure, on your own budget.

One crawler, two appetites: why blocking AI crawlers catches search too#

Here is the mechanism, and it is the whole argument in a sentence: the crawler that trains the model and the crawler that fills the index are increasingly the same animal in one coat. That is precisely why blocking AI crawlers is not a surgical strike. Where a bot is single-purpose, the block is clean and correct. Where a bot is multi-purpose, the block becomes collateral, and the collateral is your organic revenue.

Google actually offers the clean cut. Its list of common crawlers gives Google-Extended its own robots.txt token, and states plainly that Google-Extended governs whether crawled content may be used to train Gemini models and to ground responses in Gemini Apps and Vertex AI, and that it does not impact a site's inclusion in Google Search nor act as a ranking signal. Disallow that one token and you have declined the training use while keeping the index. Surgical, free, and available today in a text file.

OpenAI splits the same three ways. Its crawler documentation separates GPTBot, which feeds foundation models, from OAI-SearchBot, which surfaces sites inside ChatGPT search features, from ChatGPT-User, which fetches a page because a person asked it to. OpenAI states each setting is independent of the others. Anthropic draws the same lines in its crawler guidance: ClaudeBot for training, Claude-SearchBot for search quality, Claude-User for live user requests.

So blocking AI crawlers with a scalpel is entirely possible whenever the vendor hands you a scalpel. The trouble starts with the bots that refuse to be split into separate skins.

Cloudflare's announcement of new AI traffic options, published on 1 July 2026, sorts crawling into three purposes: Search, which collects or indexes content so it can answer questions about it later; Agent, which acts in real time on a person's behalf; and Training, which takes content to train or fine-tune a model. Where one crawler serves more than one purpose, the most restrictive rule wins.

Look at that spread and the switch-flipping instinct makes perfect sense. Tens of thousands of fetches for a single visit is not a fair forage, it is a warren being stripped. The instinct is sound and the implementation is where sites get hurt, because blocking AI crawlers at the zone level is a hammer swung in a dark den, aimed at a shape you cannot quite see.

The crawler table: who is who, and what a block really costs#

Every serious conversation about blocking AI crawlers should start with a table rather than a toggle. Print this one, take it to whoever owns the CDN, and make them say out loud which rows they intend to close. Half the risk evaporates the moment the decision is named crawler by crawler instead of category by category.

Never trust the string on its own, because anyone can forge a user agent header. Bing's webmaster team tells site owners to run a reverse DNS lookup on the requesting address, confirm the hostname ends in search.msn.com, then run a forward lookup back to the same address. Bing deliberately publishes no crawl IP ranges, because they change and hardcoded lists end up blocking bingbot by accident.

Google names the same three signals in its crawler overview: the user agent header, the source address, and the reverse DNS hostname. That page also splits Google's fleet into common crawlers, special-case crawlers and user-triggered fetchers, and notes that AdsBot can ignore the global robots.txt rule with publisher permission. Sensible ai bot management seo work starts by knowing which of those three buckets a request sits in.

The failure is silent, and silence is the expensive part#

A 403 is not a polite refusal, it is a deletion order with better manners, and it is how blocking AI crawlers turns into losing pages. Google's documentation on HTTP and network errors states that all 4xx errors except 429 are treated the same way: the crawler tells the next processing system that the content does not exist, and for Search the indexing pipeline removes the URL from the index if it was previously indexed. Read that sentence to whoever owns the edge configuration.

Search Console will tell you eventually, in a report most teams open far too rarely. The page indexing report carries a dedicated status for pages blocked by a forbidden response, and Google's guidance is direct: Googlebot never provides credentials, so a 403 aimed at it means the server is returning that error incorrectly. The remedy Google gives is to admit visitors who are not signed in, or to allow verified Googlebot requests through without a credential challenge. Blocking AI crawlers should never produce that status for a search bot.

Two comfortable myths worth a clean kill#

First, robots.txt is not an index control. Google's introduction to robots.txt says outright that it is not a mechanism for keeping a page out of Google, and a disallowed URL that other sites link to can still appear in results, simply without a description. Second, robots.txt is advisory. The standard itself, RFC 9309, published in September 2022, binds only the crawlers that choose to be bound.

There is a genuinely nasty detail buried in that RFC which almost nobody has read. If robots.txt returns a 5xx, a compliant crawler must assume complete disallow. If it returns a 4xx, the crawler may access any resource on the server. So a blanket edge rule that answers 403 to any unrecognised agent does not tighten your rules, it deletes them for every bot still willing to knock. Blocking AI crawlers badly can leave you looser than blocking nothing at all.

Crawl capacity makes the lag worse, though not quite in the way most people picture it. Google's guidance on managing crawl budget for large sites explains that server errors, rate-limiting signals and slower responses pull the crawl capacity limit down, and that crawl demand differs per crawler while the capacity itself is shared.

A forbidden response is a different beast in that respect, because Google's error documentation states that 4xx codes other than 429 have no effect on crawl rate. So the block does not slow the crawl politely, it strips the pages out while the crawler keeps knocking. Documentation on crawl budget is worth a careful read before anyone starts throttling bots at the edge, because a site that crawls slowly recovers slowly.

A five-step audit to run before 15 September 2026#

None of this needs a replatform, a retainer or a heroic sprint. It needs one afternoon, one shared document, and one person willing to ask an awkward question in a channel where nobody usually asks. Run the audit now, while blocking AI crawlers is a choice you are making rather than a default you inherited on a date somebody else set.

The reporting layer matters as much as the rules. If impressions slide and nobody can say whether the cause was an algorithm, a competitor or a checkbox, the argument in the review meeting will be settled by whoever speaks with most conviction rather than most evidence. That is how good sites lose quarters. Our work on AI visibility and citation reporting exists for exactly that reason.

There is a strategic reading here too, and it is not the doom-laden one. Being visible to AI systems is now part of the same discipline as being visible in search, which is why we treat them as one brief in SEO and GEO services rather than two competing workstreams. Blocking AI crawlers wholesale opts you out of the answer layer as surely as a noindex tag opts you out of the results page, and the pieces on SEO versus GEO and the hidden costs of opting out make the same point from the other direction.

Regulated categories carry the extra weight, as ever. An operator in iGaming marketing or FinTech marketing that vanishes from the index does not merely lose traffic, it loses the ability to correct what a model says about its licence while somebody else's page fills the gap. Sound content marketing assumes the content can be reached. Check the gate before you trust the trail.

The fox lesson, then, and it costs nothing to learn twice. Do not close the whole hedgerow. Find the gap the quarry actually uses, close that one, mark it on a map somebody else can read, and come back in September to check the gate still swings the way you left it.

Frequently asked questions#