---
title: "Shared AI training control in robots.txt: what Google plans"
description: "Google plans to support a shared AI training preference signal in robots.txt, based on IETF work. How it will work next to Google-Extended."
date: 2026-11-02
category: SEO
tags: ["robots.txt", "Google-Extended", "AI", "IETF", "AIPREF"]
url: https://uper.pl/en/blog/google-shared-ai-training-control-robots-txt/
---

# A shared AI training control in robots.txt Google wants one signal for participating AI providers

On October 30, 2026, John Mueller announced on the Google Search Central blog that Google will support a **shared AI training control in robots.txt**. Instead of adding a separate rule for every bot, a publisher will set one preference that many AI providers read at once. Google plans to roll it out in 2027 together with other AI providers. For now it is an announcement, not a finished standard that site owners can already use.

## What Google announced

The idea comes from the AIPREF working group at the IETF (Internet Engineering Task Force), the body that standardizes internet protocols. A publisher adds a shared token that expresses their preference (Google calls it a "policy token") to robots.txt and uses it to state one position toward every AI provider that signs up.

One interpretation rule matters most. If a site opts out of training with the new token and robots.txt does not name `Google-Extended` explicitly, Google will treat it as if the publisher had disallowed `Google-Extended`. Other providers can adopt the same rule, so you can object to model training by all participating providers at once or decide provider by provider.

Company-specific controls stay and work alongside the new signal. You set the broad preference with the shared token and fine-tune it for Google with `Google-Extended`. Google also points out that sites which explicitly allow `Google-Extended` are eligible to appear in Gemini app responses and get clicks from them through grounding, which means feeding the model content from the index at query time.

## How it looks in practice: a token, not a directive

The blog post shows no syntax. It turns up in Google's documentation on common crawlers instead: the `Google-Extended` entry says Google intends to support the shared, machine-readable AIPREF training preference "in early 2027", written as `user-agent: __usage_ai-training`.

The IETF draft proposes something else. The current attachment document (draft-ietf-aipref-attach-05, August 2026) describes a `Content-Usage` directive, for example:

```txt
User-Agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y
```

This blocks training across the whole site except the `/ai-ok/` directory. It is syntax from the IETF draft, not the syntax Google announced, so do not treat it as implementation instructions. Google's announcement and documentation, though, talk about a token in the `user-agent` line rather than `Content-Usage`, and Google does not explain the difference or say whether the final standard will merge the two. While the draft is still moving, neither syntax belongs on a production site.

## What "AI training" means in the IETF proposal

The group's second document, the draft vocabulary for AI content usage preferences (draft-ietf-aipref-vocab-08, September 2026), defines several categories of content use:

| Category | Key | Meaning in the draft |
|----------|-----|----------------------|
| AI training | `train-ai` | Using content to change the parameters of a generative AI model |
| AI use | `ai-use` | Using content as input to a generative model when the user does not supply it directly |
| Search | `search` | Use in an application whose main purpose is pointing users to where content is |

For now Google will use only the definition of the AI training category, and talks on the rest of the vocabulary continue. The draft itself warns that it does not reflect working group consensus and is meant as a basis for discussion, so names and definitions may still change.

## What it changes for site owners

Very little today, though the direction is clear.

- **Fewer rules to maintain.** Right now robots.txt carries a list of AI bots that needs regular updates, which I covered in the piece on [AI crawlers and search crawlers](/en/blog/ai-crawlers-vs-search-crawlers/). A shared signal will shorten that list without removing it, because not every provider will support it.
- **Company tokens stay.** If you want to treat Google differently from everyone else, `Google-Extended` is still the tool. I described its scope in the article on [what works in Google's robots.txt](/en/blog/google-robots-txt-what-you-can-put-in-it/).
- **Training and search are separate.** According to the documentation, `Google-Extended` does not affect whether you show up in search results or how you rank. Visibility in AI Overviews and AI Mode is yet another setting, covered below.
- **No retroactive effect.** According to the IETF draft, a crawler applies the preferences from the copy of robots.txt it had when it fetched a resource. New rules do not automatically cover content collected earlier; they apply only once the crawler has fetched the updated file and then fetched that resource again.

The mechanism also has a clear limit. The vocabulary draft says plainly that preferences do not guarantee anyone will honor them and do not settle whether they should. It works like a request nobody is obliged to grant, so a real block still has to be enforced on the server or at the CDN.

## Training, grounding and AI Overviews: which setting controls what

These are easy to mix up, because Google now has several switches and each covers something different.

| Setting | What it controls | What it does not control |
|---------|------------------|--------------------------|
| `Google-Extended` in robots.txt | Training of future Gemini models, plus grounding in Gemini Apps and in Grounding with Google Search on Vertex AI | Inclusion and ranking in Search |
| Search generative AI in Search Console (Settings) | Visibility in AI Overviews, AI Mode and generative features in Discover, including as a grounding source for those features | Model training and regular search results |
| Shared AIPREF token in robots.txt (from 2027) | Training preference for providers that support it; at Google it acts as a `Google-Extended` block when that token is not named in the file | Nothing yet, because it does not work yet |
| `noindex` in a meta tag or HTTP header | Whether the page appears in Search at all | Model training |

So `Google-Extended` is not just a training switch. Blocking it also stops your content from being used to answer live queries in Gemini. It does not affect AI Overviews and AI Mode, though, because those depend on the [Search generative AI control](https://support.google.com/webmasters/answer/16908024) in Search Console, available to all sites since August 31, 2026. According to the Search Console help, excluding a site from those features does not change its presence in regular results.

![The Search generative AI setting in Search Console for uper.pl: including the site's links and content in generative Search features is selected by default, with the option to exclude them below](../../assets/images/blog/search-console-search-generative-ai.webp)

You will find it in Search Console under Settings → AI controls. The screenshot shows it for uper.pl, in its default state.

## The business risk

Opting out of training has a cost that publishers rarely think about. Google ties the chance to appear in Gemini responses, and to get clicks from them, to an explicit allow for `Google-Extended`, and the same token covers grounding. Since Google plans to treat an objection made with the shared token as a `Google-Extended` block, anyone who blocks everything with one rule will probably close that route to visibility along the way. Google did not spell this out in the announcement, so think it through before you edit the file.

Google also points to the regulatory context. The proposal is consistent with the EU Code of Practice for General Purpose AI (GPAI) model providers under the AI Act, which recognizes robots.txt as a primary way for publishers to state their preferences. That may speed up adoption, but the announcement does not tell us who will join.

## What to do now

1. **Set your policy.** Decide whether you block training completely, allow it in part or leave things as they are. Without that decision no syntax will help.
2. **Review your current robots.txt.** Check which AI tokens are already there and whether the rules contradict each other. The basics are in the [robots.txt guide](/en/blog/robots-txt/).
3. **Hold off on `__usage_ai-training`.** Google promises support only in 2027, and the IETF draft keeps changing.
4. **Watch for announcements from other providers.** A shared signal is only as useful as the number of providers that honor it.

## Summary

Google wants one preference in robots.txt to replace several separate rules for bots that train AI models. The idea builds on IETF work, and the rollout with other providers is planned for 2027. `Google-Extended` stays and will work next to the shared signal. There is no final syntax yet and no list of companies that will support it, and the standard itself is still a draft without full consensus. It is a good moment to settle your own AI policy, but too early to change any files.

## Sources