OpenAI, Google, Microsoft and Anthropic Back One AI Training Opt-Out but Keep Scraping
OpenAI, Google, Microsoft and Anthropic say they will honor a single robots.txt line that lets sites refuse AI training. It is voluntary, it does not stop crawling, and half the group will not use it until 2027.
Publishers have spent years playing whack-a-mole with AI crawlers, writing a separate block for every bot that shows up. On Friday, four of the biggest AI companies promised to stop making them. Sort of.
OpenAI, Google, Microsoft and Anthropic said they will honor one shared line in a site's robots.txt file that tells them not to use its content to train AI models, Reuters reported. The catch is in the fine print. It is a preference, not a lock. Scraping stays allowed by default until a publisher changes the setting, and the setting covers only future training, according to Microsoft's blog post as relayed by Reuters.
It also does not stop bots from crawling. A site that opts out is asking companies to leave its pages out of the next model, not to stay off the site.
Two companies are early, two are not
The rollout is staggered, and the dates are the companies' own plans. Anthropic told Reuters it has "already started" implementing the signal. OpenAI said it will roll it out "in the coming months." Microsoft says it will begin on Feb. 1, 2027. Google says sometime in 2027.
The signal comes from the Internet Engineering Task Force, where Kevin Kelley, a technical lead for standards at Anthropic, proposed the shared "policy token," The Verge reported. Google's Search Central blog says the work comes out of the IETF's AI Preferences working group. It is still a draft, not an adopted internet standard. Google wrote, "We aim to support this new control along with other AI providers in 2027."
Google added a practical detail. If a site opts out with the shared token and does not mention Google-Extended, its existing AI training control, Google plans to treat that as if the publisher had disallowed Google-Extended. The older company-specific controls keep working.
OpenAI used the word "voluntary" itself.
"This voluntary effort makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once." OpenAI, in a blog post
Voluntary is the operative word, and the standards process behind this says so plainly. The IETF working group's vocabulary draft, which warns that it does not reflect working group consensus, states that a receiving party "has a choice whether to follow those preferences." It adds that preferences are not a security mechanism.
"An entity that receives usage preferences has a choice whether to follow those preferences." IETF AI Preferences working group draft
Opting out may carry a price. Microsoft and Google each said in separate blog posts that opting out of AI training could reduce the chances a publisher's content shows up in AI apps and experiences, Reuters reported. Microsoft says it will offer finer controls that separate AI training from AI grounding, which is when a model pulls from a source to answer a question, and that sites will still appear in Bing results either way, The Verge reported. Google says sites that specifically allow Google-Extended are eligible to appear in Gemini Apps responses and receive clicks.
So the deal on offer to publishers is familiar. Say no to training, and you may be asked to accept less visibility.
The pressure is real. The New York Times sued OpenAI and Microsoft in December 2023, alleging they used millions of its articles without authorization or payment to train ChatGPT, and the companies cite fair use, Reuters reported. Cloudflare has blocked AI bots by default on new customer sites since July 2025, and a standard called Really Simple Licensing lets publishers charge bots that visit.
Against that, a shared opt-out looks modest. It does nothing about content already collected. It also leaves out a major player. Meta's spokesperson Andy Stone wrote on X that a common signal "is an important step," but did not say whether Meta will honor it, The Verge noted.
Still, one signal beats a dozen, and the companies are now on record about how they plan to treat it. The test comes when OpenAI, Microsoft and Google ship their versions, and publishers can see whether their crawlers actually go quiet.
