The shared robots.txt AI-training opt-out: what the new line does, and when

Last checked 12 November 2026. General information for website owners, publishers and developers, not legal or SEO advice. The shared robots.txt token is proposed in an IETF Internet-Draft; it is not a final, endorsed IETF standard. Vendor rollout dates can slip. A robots.txt rule is a request that participating crawlers may honour; it is not a technical block. TimeBinder does not change your robots.txt for you, and a dated record is not proof that a page was kept out of a model.

What was announced on 30 October 2026

Google Search Advocate John Mueller wrote, on the official Search Central Blog, that Google plans to support a shared AI training control in robots.txt, based on work from the AIPREF working group at the IETF. The idea is that a site can express a training preference once, for every participating provider, instead of writing a separate group for each crawler. Existing company-specific controls, Google says, will continue to work alongside the joint signal.

The same day, OpenAI published “Making it easier for websites to express AI training preferences.” OpenAI says it will honour the common robots.txt signal, that this is a voluntary effort, and that it aims to implement the signal in the coming months. It is explicit that the new signal will not replace the crawler controls it already provides, such as GPTBot. Those existing controls continue to apply.

Microsoft documented the change in Bing Webmaster Tools under “New AI Preferences.” It is working with other industry participants on the shared training preference, extends that preference to individual pages (via a meta tag or HTTP header that can override robots.txt for a page), and introduces a separate page-level preference for AI Grounding — whether content can help inform AI-generated responses. Training and grounding are independent. Microsoft says these preferences do not change eligibility for traditional Bing search results. 30 October 2026 is the announcement date; Microsoft begins honouring the new preferences on 1 February 2027 as it crawls.

News outlets including The Verge and Nieman Lab reported that Anthropic is already supporting the signal, and that Anthropic engineer Kevin Kelley proposed the draft. The IETF datatracker entry for draft-kelley-aipref-training-preference-signal is an individual Internet-Draft: the IETF has not endorsed it, and it has no formal standing as a finished standard. That matters for how strongly you should treat the syntax as frozen.

What the token does, and what it does not

The draft uses an ordinary robots.txt group under the policy token __usage_ai-training. The usual example is a whole-site training opt-out, with an optional exception:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Points that are easy to get wrong:

• Training, not crawling. The token states a preference about using content to modify a generative model’s learned parameters. It does not, in the draft or in the vendor posts we read, tell a crawler to stay away. Search indexing is a different question. Microsoft is explicit that limiting training or grounding does not stop a page appearing in Bing search results. Google likewise frames this as a training control that builds on Google-Extended, not as a change to Search rankings.
• A wildcard group does not count. In the draft, a User-Agent: * group never matches the token. A robots.txt file with no __usage_ai-training group has expressed no preference. Rules you already wrote for GPTBot, Google-Extended, ClaudeBot or others are still useful, and they are not automatically read as this shared preference.
• Defaults and overrides. Microsoft says Allow is the default for both training and grounding when no preference is published. A page-level Microsoft training tag can override robots.txt for that page, for Microsoft only. Private agreements between a site and a company can also override the public signal, according to reporting on the draft.
• It does not un-train anyone. A new preference applies to future training on the timetable each company states. It does not take older crawls out of a model already trained.
• Grounding is a separate switch, and not always in robots.txt. Google says that if a site opts out with the shared token and does not specify Google-Extended, Google plans to treat that as if the publisher had disallowed Google-Extended. Google-Extended also covers whether content can be used for grounding in Gemini Apps and on Vertex AI; sites that specifically allow Google-Extended are eligible to appear in Gemini Apps responses through grounding. Microsoft’s grounding preference is page-level only (Allow, Disallow, or DisallowExtended) and cannot be set in robots.txt. Opting out of training is not automatically opting out of AI answers that cite you, or vice versa.

When each company says it will honour the signal

• Anthropic. Reported by The Verge and Nieman Lab as already supporting the signal. We did not treat a third-party report as a substitute for Anthropic’s own documentation; check Anthropic’s current crawler pages before you rely on a date.
• OpenAI. Official post: implementation “in the coming months,” with existing GPTBot and other robots.txt controls still in effect.
• Google. Official blog: Google aims to support the control in 2027, “along with other AI providers,” and will provide updates.
• Microsoft. Official help: honours the new preferences from 1 February 2027 as it crawls. For sites that used NoArchive or NoCache as training or grounding controls, Microsoft sets a transition: publish the new training preference by 1 March 2027 (after that date, NoArchive and NoCache will not be honoured for training); grounding has additional temporary settings in Bing Webmaster Tools, with a 15 December 2026 / 1 March 2027 / 30 September 2027 timetable described on that help page.

Until a company actually honours the shared token, keep the per-crawler rules you already use. Our free AI crawler opt-out generator is for those existing robots.txt and TDM rules. It is a request list, not a block, and it does not replace reading each vendor’s current documentation.

Practical checklist

1. Decide what you actually want to say about training, about appearing in AI answers, and about ordinary search. Those are not one switch.
2. Add a __usage_ai-training group if you want to express a shared training preference, using Allow and Disallow paths you mean. Do not assume a * group already covers it.
3. Keep your per-bot rules (GPTBot, Google-Extended, and others) at least until each vendor’s rollout date, and probably after, because company-specific controls remain valid.
4. If you used NoArchive or NoCache for Microsoft training or grounding, read Microsoft’s transition dates and do not assume the old tags will keep doing that job after early 2027.
5. Keep a dated copy of the robots.txt you published (and of any page-level meta tags). A TimeBind can record that this exact file existed on a given date and still matches. That is useful if you later need to show what preference you had posted. It does not prove a crawler honoured it, and it does not prove a page was left out of a model.

If you publish work on a site you do not control, a robots.txt rule on someone else’s domain will not speak for you. See how to protect art from AI scraping when you post it online and Glaze, Nightshade and official opt-outs.

What a TimeBind does and does not show

A TimeBind is a dated, checkable record that an exact file existed no later than a recorded time and still matches. The SHA-256 fingerprint is made in your browser and the file is never uploaded. The record is anchored to Bitcoin (a Merkle root in an OP_RETURN). Your first Pooled TimeBind is free; after that, TimeBinds start from US$5, and an ASAP TimeBind is US$15. See how it works.

Here, the useful files are often the robots.txt you actually served, a screenshot of a webmaster tool, or the articles and images you are trying to protect. You can later show that this exact file existed no later than the recorded time and still matches.

A TimeBind does not add a line to robots.txt, does not bind a crawler, does not show whether a page was used in training, and does not change search rankings.

Limits

The syntax lives in a draft. Vendors say they will follow it and will give notice if it changes, but it is not a finished IETF standard. Support rolls out on different calendars, some of them in 2027. Scrapers that ignore robots.txt will ignore this token too. An opt-out of training is not automatically an opt-out of AI citations, and neither is a licence, a payment, or a guarantee of traffic. Google’s post notes that the approach is consistent with the EU Code of Practice for General-Purpose AI model providers under the EU AI Act, which recognises robots.txt as a way for publishers to express machine-readable preferences — that is a policy context, not a promise that every model provider in the EU already honours this exact token.

Tools that can help

Start with the official posts: Google Search Central, OpenAI, and Microsoft’s New AI Preferences help. On this site, the free AI crawler opt-out generator helps you draft per-crawler robots.txt and TDM rules, and the opt-outs guide for artists covers other official mechanisms. For a dated copy of a file you publish, see how a TimeBind works.

For short definitions of text and data mining (TDM) and AI training opt-out, see the glossary.

Sources checked

• John Mueller, “Supporting a shared AI training control for websites,” Google Search Central Blog, 30 October 2026, https://developers.google.com/search/blog/2026/10/ai-training-control
• OpenAI, “Making it easier for websites to express AI training preferences,” https://openai.com/index/helping-websites-express-ai-training-preferences/ (read 12 November 2026)
• Microsoft Bing Webmaster Tools, “New AI Preferences,” https://www2.bing.com/webmasters/help/new-ai-preferences-2t1cc16d (read 12 November 2026)
• IETF Datatracker, draft-kelley-aipref-training-preference-signal, https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/ (individual Internet-Draft; not an endorsed IETF standard; read 12 November 2026)
• The Verge, “The biggest names in AI are backing a new way to opt out of scraping,” https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out (read 12 November 2026)

Date the robots.txt you actually published

Your first Pooled TimeBind is free, with no credit card required. After that, TimeBinds start from US$5, and an ASAP TimeBind is US$15. Click here to start, or see plans and pricing.

Related: VSCO says it will not train generative AI on your photos: what creators should keep.