Overview: AI Training Opt Out
Understand the AI training opt out signal, the companies’ rollout timelines and why a crawler preference is different from a technical block.
Google, OpenAI, Microsoft and Anthropic Agree on One robots.txt Rule to Opt Out of AI Training. There Is a Catch.
TL;DR
- On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will honor one shared
robots.txtsignal,__usage_ai-training, as an opt-out from AI training.- Only Anthropic has started building support. OpenAI says “the coming months,” Google says 2027 (its crawler documentation says early 2027), and Microsoft starts on February 1, 2027.
- The signal states a preference. It does not stop crawling, private deals can override it, and the IETF draft behind it has no formal standing yet.
- Do not delete your existing
GPTBot,Google-ExtendedorClaudeBotrules. Add the new group next to them.
Reviewed by Imran Khan Pathan, Editor at AI Tech Safar. We checked this article against the text of the IETF draft itself, the Google Search Central and Microsoft Bing announcements, and reporting from The Next Web and Search Engine Roundtable. Four things were corrected from our first draft: Kevin Kelley’s job title (we could not confirm it, so we removed it), the Cloudflare claim (default blocking applies to new domains), the description of RSL (it lets publishers state licensing terms but cannot enforce them on its own), and our advice on when to add the new token.
Published and last updated: November 1, 2026. We will note any correction here.
For roughly three years, opting out of AI training has meant keeping a list. GPTBot for OpenAI. Google-Extended for Google. ClaudeBot for Anthropic. Then a long tail of smaller crawlers. Every new AI lab brought a new user-agent string and one more entry in the robots.txt file.
On October 30, 2026, four of the largest Western AI companies said they would stop making publishers do that. Google, OpenAI and Microsoft each published an announcement that day, and Anthropic’s Kevin Kelley posted the underlying IETF draft and introduced it on the IETF’s AI preferences mailing list. All four will honor a single shared signal, a policy token named __usage_ai-training.
The headline reads like a win for the open web. The details are more mixed: the start dates are months apart, and the signal only asks politely.
What the Signal Actually Does
The token is not a new file format and it does not add a new directive. It is a name that goes on the User-Agent line of an ordinary robots.txt group. The publisher then lists paths with the usual Disallow and Allow rules. The example in the draft blocks AI training everywhere except a press-kit folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/A Disallow rule expresses a preference against AI training for that path. An Allow rule expresses a preference in favor. If the file has no group for the token, the site has expressed no preference at all.
The draft comes from Kevin Kelley of Anthropic and was posted as an individual Internet-Draft on October 30. It borrows its definitions from the IETF’s AI Preferences (AIPREF) working group. Under those definitions, “AI training” means using content to change the learned parameters of a generative AI model. Training a model that only does non-generative work, such as classification, is outside the definition.
The practical gain is obvious. One block can speak to several companies at once. Google’s own announcement says the goal is to let sites “express their AI training preferences across multiple AI providers in a single step.”
Who Starts When
The four companies did not agree on a shared timeline, and that matters more than the announcement itself.
| Company | Status | What It Means for Publishers |
|---|---|---|
| Anthropic | Already started building support | The company behind the draft is moving first. Kelley warned that adding support across its services will take time and surface edge cases. The token will sit alongside Anthropic’s existing crawler controls, not replace them. |
| OpenAI | “Coming months” | No fixed date. Existing GPTBot controls keep working in the meantime. |
| 2027 (crawler documentation says early 2027) | Until then, Google-Extended is the only control that works for Google’s AI training. Once support arrives, a site that uses the token without mentioning Google-Extended will be treated as having blocked it. | |
| Microsoft | February 1, 2027 | Replaces the older NoArchive and NoCache tags for training, and adds a separate control for AI grounding. |
For roughly the next three months, then, the shared signal is mostly a promise. A publisher who adds it today gets Anthropic’s support and not much else yet.
Microsoft Splits Training From Grounding
Microsoft went further than the other three. Its Bing team published a separate set of controls the same day, built on the shared training token. The key change is that training and grounding are now independent choices. Grounding is the step where an AI answer pulls in a live web page; training changes the model itself.
Bing now offers three settings: AI Training, AI Grounding, and Limited Grounding. The limited option lets Microsoft’s own products, such as Copilot, use a page while blocking the grounding services Microsoft sells to other companies. If you publish no preference, both training and grounding default to allowed. Blocking grounding does not change where a page ranks in Bing search, but Microsoft says it will significantly reduce a page’s visibility in AI-generated answers.
Krishna Madhavan, a principal product manager at Bing, put the reasoning plainly: publishers should not have to choose between being found in Bing and deciding how generative models use their work.
The dates matter if you rely on the old tags:
| Date | What Happens |
|---|---|
| December 15, 2026 | Temporary grounding controls appear in Bing Webmaster Tools for eligible sites that use NoArchive or NoCache. |
| February 1, 2027 | Microsoft starts honoring the new preferences as it crawls. |
| March 1, 2027 | Deadline to set the new preferences. After this date, NoArchive and NoCache are no longer read as a training or grounding preference. |
| September 30, 2027 | The temporary grounding preferences expire. |
The Catch: It Is a Preference, Not a Block
The draft is explicit that the token is not a directive for crawling behavior. It tells a company how a site would like its content used. It does not stop a crawler from fetching the page, and nothing technical enforces it. Compliance rests on each company keeping its word. A new lab that never signed on has no obligation to read it.
Three limits are written into the announcements and the draft itself:
- Private deals can override the signal. The draft says private agreements may override a preference expressed this way. If you license content to an AI company, that contract wins over your
robots.txt. - It creates no licensing market. Microsoft said the new preferences do not establish licensing or payment arrangements. This is an opt-out, not a way to get paid. Anyone who wants compensation still has to negotiate it separately.
- It is not a finished standard. The IETF datatracker states that the draft is not endorsed by the IETF and has no formal standing in the standards process. Drafts are valid for a maximum of six months, and this one expires on May 3, 2027 unless it is updated or replaced.
Where This Fits in the Wider Scraping Fight
Two other approaches try to push harder than a polite request.
Cloudflare has blocked AI crawlers by default on new domains since July 2025, and it launched a Pay Per Crawl system in private beta that lets publishers charge for access. Because Cloudflare sits in front of the site, its blocks are enforced, not requested.
The Really Simple Licensing (RSL) standard is closer to the new token in spirit. Its 1.0 specification arrived in December 2025 with support from Cloudflare and Akamai, and it lets publishers publish machine-readable licensing terms, including pay-per-crawl. On its own, though, RSL cannot stop a crawler. It needs infrastructure partners to enforce it.
The new training signal sits on the voluntary end of this spectrum. It is cheap for AI companies to support, it does not commit them to paying for anything, and it gives publishers less leverage than a CDN-level block. Google also framed it as consistent with the EU’s Code of Practice for general-purpose AI under the AI Act, which treats robots.txt as a main way for publishers to state machine-readable preferences.
Meta is not part of the announcement. Its communications head, Andy Stone, voiced support for a common signal on X, according to The Verge, but he did not say whether Meta would honor this one.
What You Should Do Right Now
Keep every crawler-specific block you already have. The new token is built to sit beside them, and until Google, OpenAI and Microsoft switch it on, those older lines are still the only controls that work for them.
Adding the new group today costs nothing. The draft was designed so that parsers that do not recognize the token simply ignore the group, so your existing rules behave as before. A file that covers both approaches looks like this:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-Agent: __usage_ai-training
Disallow: /Two follow-ups are worth doing. If your site uses Microsoft’s NoArchive or NoCache tags, set the new Bing preferences before March 1, 2027. And decide whether you want to block grounding as well as training, because blocking grounding trades away visibility in AI-generated answers.
What We Still Don’t Know
- OpenAI’s exact date. “Coming months” is the only commitment so far.
- Whether other companies join. Meta has voiced support for a common signal but has not said it will honor this one, and the announcement names only four companies.
- Whether the IETF adopts the draft. It is an individual submission today. A working group could change it substantially.
- How compliance will be checked. Neither the draft nor the announcements describe any way for a publisher to verify that a company is honoring the signal.
Our Read
The useful part of this announcement is the plumbing. One group in robots.txt is easier to maintain than a list that grows every time a lab launches a crawler, and having Google, OpenAI, Microsoft and Anthropic all point at the same wording removes a real source of confusion.
The weaker part is everything around it. The commitments are voluntary, the dates are spread across 2027, and the document underneath is a draft with no standing. For a small publisher the sensible move is modest: add the group, keep your old rules, and treat it as one layer of protection rather than a lock.
FAQ
Will adding the token hurt my Google rankings or get pages dropped from search?
The draft treats content-use preferences and crawling rules as independent, so the token alone is not a crawling instruction. Google has said it will read the token as a block on Google-Extended, which covers Gemini training and leaves the page in Google Search. Microsoft says blocking its grounding setting does not change a page’s Bing ranking, though it can cut visibility in AI-generated answers. None of this is live yet, so there is no real-world ranking data to point to.
I publish on Blogger. Where do I add this?
In Blogger, open Settings, go to the crawlers and indexing section, and turn on custom robots.txt. Paste in your existing rules first, including any sitemap line, and add the new group underneath. Menu labels can change between Blogger updates. Afterward, open yourdomain.com/robots.txt in a browser to confirm the new group is showing.
Does a blanket “block everything” rule already opt me out under this system?
No. The draft says a wildcard group, User-agent: *, must not be treated as matching the token. A site with only wildcard rules has expressed no AI-training preference. The token needs a group of its own.
Will it remove my content from models that already trained on it?
Nothing in the draft pulls back content that was already collected. A preference attaches to the robots.txt file that was in effect when a page was retrieved. Microsoft likewise describes its settings as applying to future model training.
What is the difference between AI training and AI grounding, and why does it matter for traffic?
Training uses your content to change a model’s parameters. Grounding fetches your page at the moment someone asks a question, so the answer can reference it. Blocking training keeps your writing out of future models. Blocking grounding keeps it out of live answers, which is where AI products may cite and link to you. Microsoft is the first of the four to give these separate switches.
How can I tell whether a company is honoring the signal?
You cannot verify it directly. Server logs can show whether named crawlers such as GPTBot or ClaudeBot keep fetching your pages, but they cannot show what happens to the content afterward. The draft also says the token is not a crawling directive, so a company may still fetch a page for some other purpose. Google has promised progress updates on its rollout.
Is it worth bothering with if I run a small blog?
It takes a few lines and, by design, should not affect how existing crawlers read your file. The real effect depends on each company’s compliance, and it will not earn you any money. If you object to your writing being used for training, it is a cheap extra layer. If you want to appear in AI answers, think carefully before blocking grounding.
Related Reading on AI Tech Safar
- OpenAI’s Rogue AI Agents Accessed Crypto Wallets – As Anthropic Reveals $80 Billion Loss
- OpenAI’s Rogue AI Agents Went Further Than Anyone Knew — Now It’s Spending $500,000 a Day to Find Out
- AI Agent Security: The Complete 2026 Guide to Protecting Against Rogue AI
Useful Sources
- IETF Datatracker — Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)
- Google Search Central — Supporting a shared AI training control for websites
- Microsoft Bing Blog — Your content, your choice: clearer preferences for training and grounding
- OpenAI — announcement on website AI training preferences
- The Next Web — Google, Microsoft, OpenAI and Anthropic back one AI training opt-out
- Search Engine Roundtable — Microsoft Bing Separates AI Training From AI Search Grounding With New Controls
- The Verge — report on the shared AI scraping opt-out
- RSL Standard — Really Simple Licensing
For the broader context and related reading, explore our Frontier AI Security: Sandboxes, Breaches and Testing.


