Meta's quiet clause: AI web scraping, now with a contract behind it

What the clause says about ai web scraping#

The fox does not trust a tidy hedgerow, and neither should a marketer reading a notice of changed terms. The notice here is short, the consequence is not. Nieman Lab reports that, starting 30 October, business accounts will consent to Meta scraping their websites if they continue using Facebook, Instagram or WhatsApp. The same report is plain about the other half: there is no option for business accounts to opt out ahead of that date.

That is the quiet clause. It does not announce itself as ai web scraping, though that is the practice it enables, and Ars Technica notes that there is no mention of AI in it at all. The training purpose is inferred from the verb "train" and from the language of automated crawling. In other words, a contract has been written that covers the activity without ever naming it, which is precisely how a thicket of boilerplate grows. Call it ai web scraping or call it indexing, the contract does not choose between them, and a reader who wants certainty will not find it in the wording.

Meta disputes the framing. A spokesperson told The Verge that there is no change to how Meta uses web crawlers, and that any suggestion to the contrary is false. Andy Stone, Meta's communications chief, said in a post that the update exists to pull accurate information about products for sale on third-party websites. Both statements can be true in their own register, and still leave a business account holder with a contract that reads more widely than either.

A small account, a fast summary, and an accurate one in the one respect that matters: the practitioner conversation on X moved quickly. The folkfox trend desk counted 48 posts on the topic in the eight hours before this article was written, which is 3.86 times the week's hourly baseline. That is a volume spike among practitioners, not a viral moment, and the chart below shows it at its true size.

Treat the spike as a reason to read the terms, not as evidence about their meaning. What the terms mean is a legal question, and the legal experts quoted by Nieman Lab disagree with Meta's reassurance in places. Nobody outside Meta has published the clause's definition of a site "associated" with a business account either, and Meta declined to answer Nieman's questions on it.

What robots.txt can and cannot do about ai scraping#

Meta's answer to the worry is robots.txt. The spokesperson quoted by Nieman Lab said content owners can still manage how their content is accessed by using robots.txt, and Meta's own crawler page says the way to stop a crawler is to add a disallow for it. That is useful against ai web scraping you want to stop, and it is also narrower than it sounds.

Start with what Meta documents. Its Meta Web Crawlers page describes a training crawler that crawls the web for use cases such as training foundation AI models or improving products by indexing content directly, and a second crawler that navigates the web to improve Meta AI search result quality. They have different tokens, so blocking one does not block the other. Changes also lag, because crawlers may cache the contents of robots.txt for up to 24 hours.

Then look at what robots.txt is. RFC 9309, the Robots Exclusion Protocol standard, states that its rules are not a form of access authorisation. Google Search Central says much the same in its own terms: the file is not a mechanism for keeping a web page out of Google. A robots file is a request that polite crawlers honour, which is exactly why lawyers asked whether a commercial contract could be read to sit above it.

The third row deserves attention from anyone who runs paid social. Meta's FacebookExternalHit documentation says that crawler might bypass robots.txt when performing security or integrity checks. A site owner cannot treat the file as a single master switch for everything Meta fetches, and should not tell a board that it is.

The legal ground under meta commercial terms#

The patient prowl through the law starts with a modest fact. Nobody has tested this clause in court. Everything below is reasoning from public documents, not a ruling, and Meta's live October wording was not available to us in full: the commercial terms page we could open still showed the March 2024 text, so the clause as quoted comes from the three news outlets that reproduce it.

Even so, the structure of the argument is worth knowing. Under the EU's Digital Single Market copyright directive (2019/790), the more general text and data mining exception in Article 4 only applies if the copyright holder has not opted out. That is the secondary summary on Wikipedia, not the Official Journal text, which we could not read in full on the day, so check EUR-Lex before relying on it. The tension lawyers are circling is plain enough: a contract that authorises ai web scraping sits awkwardly beside a rightholder's opt-out that says the opposite.

The wording of Meta's reassurance matters too. The company says there is no change to how it uses crawlers and that robots.txt still applies. If that is the whole truth, the clause adds little in practice to ai web scraping for sites that already disallow the training crawler. If it is not, the clause adds a great deal for sites that did not. AI Weekly's summary of the coverage lands on the same plain reading: there is no opt-out, only a decision about whether to keep using the products.

That decision is spelled out in the pre-update terms preserved by ConductAtlas: if you do not agree to the updated terms, please stop all access or use of our Products for business or commercial purposes. For a brand spending real money on Meta, that sentence turns a copyright debate into a budget conversation, and it is why the practical checks in the next section come before any lobbying.

Five checks before 30 October#

The sensible reply to a contract you cannot negotiate, and to ai web scraping you cannot veto, is housekeeping. None of the checks below needs a lawyer to start, none costs more than an afternoon, and together they turn a vague worry about ai web scraping into a record. On the day of writing there are 13 days left.

• 30 Oct

  terms take effect

• 13

  days left on 17 October

• 24 h

  crawler cache for robots.txt

• 0

  documented business opt-out routes

Verification is the quiet hinge. Meta's domain verification documentation says to add a DNS TXT entry to your DNS record to confirm that you own the domain, and any one of the methods is enough. Verifying a domain does not change the clause or stop ai web scraping, but it tells you exactly which of your sites Meta already recognises as yours, and that list is the one to audit.

A word on the fourth check. Documents drift, and a screenshot of the terms with a date on it costs nothing. The fox leaves tracks on purpose so that it can prove later where it has been.

What it means for the ad budget and the brand#

There is an irony in the commercial layout that is worth saying aloud. The Verge reports that Meta launched a Meta One subscription last month that requires businesses and creators to pay to include links in posts and reels. A business can therefore pay Meta to carry a link to its site, and under the new terms agree to let Meta crawl the site behind the link.

That is a negotiation point for agencies and large advertisers who have account managers, and a procurement point for everyone else. Ask the account team to put Meta's explanation of the clause in writing, because the spokesperson's statement to the press is not a contract term. Ask also how a site is judged to be associated with a business account, since that definition is the one thing nobody has published.

For brands, the quieter risk is content. In plain terms, ai web scraping of a site full of original reporting, research or product photography is exactly what a training crawler wants most, and the licensing value of that work is what the legal experts were pointing at. Lawyers call the practice AI data scraping when the purpose is training, and it is the commercial value of that data, not the crawling itself, that makes the clause worth reading twice. Our earlier look at Meta AI ads and the guardrails that stay yours made a similar argument about handing the machine the keys: delegate the drudgery, never the terms. The same logic holds for brand salience and the shopping agent, where the question is who gets to describe your brand to a model.

If your paid social sits on a site with commercial content you would never license cheaply, folkfox paid social and SEO and GEO work should start from the same inventory: what is public, what is protected, and what a crawler is told. The brand layer of that is a brand strategy decision before it is a technical one, and AI consultancy is where the model-training question properly sits.

Frequently asked questions#