A shared AI training control in robots.txt Google wants one signal for participating AI providers

On October 30, 2026, John Mueller announced on the Google Search Central blog that Google will support a shared AI training control in robots.txt. Instead of adding a separate rule for every bot, a publisher will set one preference that many AI providers read at once. Google plans to roll it out in 2027 together with other AI providers. For now it is an announcement, not a finished standard that site owners can already use.

What Google announced

The idea comes from the AIPREF working group at the IETF (Internet Engineering Task Force), the body that standardizes internet protocols. A publisher adds a shared token that expresses their preference (Google calls it a “policy token”) to robots.txt and uses it to state one position toward every AI provider that signs up.

One interpretation rule matters most. If a site opts out of training with the new token and robots.txt does not name Google-Extended explicitly, Google will treat it as if the publisher had disallowed Google-Extended. Other providers can adopt the same rule, so you can object to model training by all participating providers at once or decide provider by provider.

Company-specific controls stay and work alongside the new signal. You set the broad preference with the shared token and fine-tune it for Google with Google-Extended. Google also points out that sites which explicitly allow Google-Extended are eligible to appear in Gemini app responses and get clicks from them through grounding, which means feeding the model content from the index at query time.

How it looks in practice: a token, not a directive

The blog post shows no syntax. It turns up in Google’s documentation on common crawlers instead: the Google-Extended entry says Google intends to support the shared, machine-readable AIPREF training preference “in early 2027”, written as user-agent: __usage_ai-training.

The IETF draft proposes something else. The current attachment document (draft-ietf-aipref-attach-05, August 2026) describes a Content-Usage directive, for example:

User-Agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y

This blocks training across the whole site except the /ai-ok/ directory. It is syntax from the IETF draft, not the syntax Google announced, so do not treat it as implementation instructions. Google’s announcement and documentation, though, talk about a token in the user-agent line rather than Content-Usage, and Google does not explain the difference or say whether the final standard will merge the two. While the draft is still moving, neither syntax belongs on a production site.

What “AI training” means in the IETF proposal

The group’s second document, the draft vocabulary for AI content usage preferences (draft-ietf-aipref-vocab-08, September 2026), defines several categories of content use:

Category	Key	Meaning in the draft
AI training	train-ai	Using content to change the parameters of a generative AI model
AI use	ai-use	Using content as input to a generative model when the user does not supply it directly
Search	search	Use in an application whose main purpose is pointing users to where content is

For now Google will use only the definition of the AI training category, and talks on the rest of the vocabulary continue. The draft itself warns that it does not reflect working group consensus and is meant as a basis for discussion, so names and definitions may still change.

What it changes for site owners

Very little today, though the direction is clear.

• Fewer rules to maintain. Right now robots.txt carries a list of AI bots that needs regular updates, which I covered in the piece on AI crawlers and search crawlers. A shared signal will shorten that list without removing it, because not every provider will support it.
• Company tokens stay. If you want to treat Google differently from everyone else, Google-Extended is still the tool. I described its scope in the article on what works in Google’s robots.txt.
• Training and search are separate. According to the documentation, Google-Extended does not affect whether you show up in search results or how you rank. Visibility in AI Overviews and AI Mode is yet another setting, covered below.
• No retroactive effect. According to the IETF draft, a crawler applies the preferences from the copy of robots.txt it had when it fetched a resource. New rules do not automatically cover content collected earlier; they apply only once the crawler has fetched the updated file and then fetched that resource again.

The mechanism also has a clear limit. The vocabulary draft says plainly that preferences do not guarantee anyone will honor them and do not settle whether they should. It works like a request nobody is obliged to grant, so a real block still has to be enforced on the server or at the CDN.

Training, grounding and AI Overviews: which setting controls what

These are easy to mix up, because Google now has several switches and each covers something different.

Setting	What it controls	What it does not control
Google-Extended in robots.txt	Training of future Gemini models, plus grounding in Gemini Apps and in Grounding with Google Search on Vertex AI	Inclusion and ranking in Search
Search generative AI in Search Console (Settings)	Visibility in AI Overviews, AI Mode and generative features in Discover, including as a grounding source for those features	Model training and regular search results
Shared AIPREF token in robots.txt (from 2027)	Training preference for providers that support it; at Google it acts as a Google-Extended block when that token is not named in the file	Nothing yet, because it does not work yet
noindex in a meta tag or HTTP header	Whether the page appears in Search at all	Model training

So Google-Extended is not just a training switch. Blocking it also stops your content from being used to answer live queries in Gemini. It does not affect AI Overviews and AI Mode, though, because those depend on the Search generative AI control in Search Console, available to all sites since August 31, 2026. According to the Search Console help, excluding a site from those features does not change its presence in regular results.

You will find it in Search Console under Settings → AI controls. The screenshot shows it for uper.pl, in its default state.

The business risk

Opting out of training has a cost that publishers rarely think about. Google ties the chance to appear in Gemini responses, and to get clicks from them, to an explicit allow for Google-Extended, and the same token covers grounding. Since Google plans to treat an objection made with the shared token as a Google-Extended block, anyone who blocks everything with one rule will probably close that route to visibility along the way. Google did not spell this out in the announcement, so think it through before you edit the file.

Google also points to the regulatory context. The proposal is consistent with the EU Code of Practice for General Purpose AI (GPAI) model providers under the AI Act, which recognizes robots.txt as a primary way for publishers to state their preferences. That may speed up adoption, but the announcement does not tell us who will join.

What to do now

1. Set your policy. Decide whether you block training completely, allow it in part or leave things as they are. Without that decision no syntax will help.
2. Review your current robots.txt. Check which AI tokens are already there and whether the rules contradict each other. The basics are in the robots.txt guide.
3. Hold off on __usage_ai-training. Google promises support only in 2027, and the IETF draft keeps changing.
4. Watch for announcements from other providers. A shared signal is only as useful as the number of providers that honor it.

Summary

Google wants one preference in robots.txt to replace several separate rules for bots that train AI models. The idea builds on IETF work, and the rollout with other providers is planned for 2027. Google-Extended stays and will work next to the shared signal. There is no final syntax yet and no list of companies that will support it, and the standard itself is still a draft without full consensus. It is a good moment to settle your own AI policy, but too early to change any files.

Frequently Asked Questions

Does the shared AI training control work yet?

No. Google announced on October 30, 2026, that it intends to support it and is aiming for 2027, together with other AI providers. For now only company-specific tokens such as Google-Extended work.

Will the new signal replace Google-Extended?

No, it will work alongside it. What matters is whether robots.txt contains a separate rule for Google-Extended. If you opt out of training with the shared token and there is no such rule, Google will treat that as disallowing Google-Extended. If there is one, you can use it to set Google differently from the other providers.

Will blocking AI training affect my Google rankings?

According to the documentation, Google-Extended does not affect search inclusion or ranking. Visibility in AI Overviews and AI Mode is governed by the separate Search generative AI setting in Search Console. Google does, however, tie allowing Google-Extended to appearing in Gemini responses through grounding, so a block may close that route.

Do AI providers have to honor the signal?

The standard does not enforce it, and the IETF draft states that preferences do not guarantee compliance. The EU GPAI Code of Practice may add pressure on providers, but technically this is a voluntary signal, not a block.

Sources

1. Google Search Central Blog: Supporting a shared AI training control for websites https://developers.google.com/search/blog/2026/10/ai-training-control
2. Google Search Central: Google common crawlers (Google-Extended) https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
3. Google Search Console Help: Search generative AI control https://support.google.com/webmasters/answer/16908024
4. IETF: AI Preference Attachment (draft-ietf-aipref-attach-05) https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.html
5. IETF: A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08) https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/