Plumbing

Opting out of AI training? At Google, that opts you out of Gemini's citations too.

By Arnav Mukherjee, founder of TofuBofu · October 31, 2026

Every scan we run asks Gemini with Google Search grounding switched on, because that's how Gemini finds pages it can cite. So when Google posted on Friday about a new robots.txt control, I read it for one thing: whether it changes who Gemini can cite.

It does. The headline is a shared "no AI training" signal that Google, OpenAI and any provider that adopts it will read. The fine print is that at Google, the signal lands on a token that also decides whether Gemini can ground its answers in your pages. A founder who ticks "no training" because it sounds safe could drop out of Gemini's citations in 2027 and never connect the two.

What did Google and OpenAI announce on 30 October?

Google's Search Central blog, posted by John Mueller, says Google will support "a shared AI training control in robots.txt", built on work from the AIPREF working group at the IETF. The aim is one line that reaches every participating AI provider at once, instead of a separate rule per company. Google says it aims to support it "along with other AI providers in 2027". Its crawler documentation says early 2027.

OpenAI published the same day. Its own summary says it "will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers." Its full post wouldn't load for us, so that summary is all I'm quoting.

The syntax comes from the IETF draft Google links to. It's an ordinary robots.txt group with a special name on the User-Agent line:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example, taken from the draft, says no training anywhere except the press kit. The draft adds no new directives and says the token "is not a directive for crawling behavior". It states a preference about use, while your crawler rules stay exactly as they were.

Why does a training opt-out reach Gemini answers?

Because of how Google wired it. Google's post says that when a site opts out of training with the token, "and Google-Extended is not explicitly specified", Google plans to treat it "as if the publisher had disallowed Google-Extended."

Google-Extended isn't only a training switch. Google's crawler documentation says it covers training future Gemini models and grounding, meaning content from the Google Search index handed to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI. The same post spells out the other side: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

So at Google, training and grounding share one token. You can't say yes to one and no to the other. A blanket "no AI training" line, read the way Google says it will read it, also removes your pages from what Gemini can ground on and link to.

Where one "no AI training" line lands __usage_ai-training Disallow: / Google reads it as Google-Extended: disallow Gemini training: out Gemini grounding: out Not touched by the token Google Search ranking Googlebot AI Overviews, AI Mode search index, nosnippet ChatGPT search OAI-SearchBot, nothing said Google Search Central blog and crawler docs, IETF draft, OpenAI RSS summary, read 31 October 2026.

Red is what Google says the token reaches. Green is what Google says it doesn't. Blue is a gap in what OpenAI has published.

What does the token leave alone?

The draft also lets a specific rule beat a general one, and says providers "MAY document" that their own token outranks the shared one. Google already has: name Google-Extended yourself and the shared line no longer speaks for you at Google.

See whether Gemini names you today. Our free scan asks real buying questions across five AI engines, Gemini included with Google Search grounding on, and shows who gets named instead of you.

Check your AI visibility free

What should a B2B firm put in robots.txt?

My position, and it's a judgement rather than a measurement: if you sell services or software to businesses, add nothing. Training is how a model learns that your firm exists and what it's good at. A buyer asking for a shortlist gets names from that memory and from search. Opting out of training trades a real recommendation channel for a protection you mostly don't need. Your pricing page isn't a novel.

There are firms with a real reason to opt out: paid research, gated benchmarks, proprietary playbooks you sell. For them, the draft's path rules are the right tool. Opt out the paid library, and tell Google explicitly that grounding is still welcome:

# No AI training on the paid library, for providers that adopt the shared token
User-Agent: __usage_ai-training
Disallow: /research/

# Google: name Google-Extended so the shared line doesn't speak for you
User-Agent: Google-Extended
Allow: /

Read that second group carefully. Under robots.txt rules, a named group replaces the wildcard group for that token. If your User-agent: * group disallows paths, copy those Disallow lines under Google-Extended or this Allow can re-open them to Gemini. And because Google-Extended is one token, allowing it lets Google train Gemini on those pages too. That's the price of staying in Gemini's grounding, and Google's documentation offers no split today. If you'd rather keep the library out of Gemini entirely, add Disallow: /research/ under Google-Extended and accept that those pages won't ground Gemini answers.

Three checks before 2027 arrives:

  1. Search your robots.txt for Google-Extended and for __usage_ai-training. If a plugin, a CDN preset or a well-meaning developer added either, decide whether you meant it.
  2. Check what your CMS or CDN will offer. My guess is that one-click "block AI training" toggles will adopt the shared token. Find out whether yours writes a Google-Extended line alongside it.
  3. Keep OAI-SearchBot, Claude-SearchBot and Googlebot allowed. They're the crawlers that find pages to cite for ChatGPT, Claude and Google. At Google, Googlebot does the fetching and Google-Extended decides training and grounding use. Claude's search providers add a twist: Brave won't crawl what Googlebot can't.

The standard is a good idea. One line beats a dozen per-company rules, and publishers asked for it. The trap is the word "training". It sounds like something that happens in a lab, far from your pipeline. At Google, the switch it maps to already decides whether Gemini can cite you today.

Frequently asked questions

What is __usage_ai-training?

It's a policy token defined in an IETF draft from the AIPREF work. You write it on a User-Agent line in robots.txt, then use ordinary Allow and Disallow rules under it. A Disallow says you prefer your content isn't used for AI training on those paths. It introduces no new directives and doesn't change how crawlers read the rest of the file.

When does Google support the shared AI training opt-out?

Google's Search Central blog, posted 30 October 2026 by John Mueller, says it aims to support the control along with other AI providers in 2027. Google's crawler documentation says early 2027. Until then, Google-Extended remains the control Google reads.

Does opting out of AI training remove me from Google Search or AI Overviews?

No. At Google the token maps to Google-Extended, and Google's documentation says Google-Extended does not impact a site's inclusion in Google Search and isn't a ranking signal. AI Overviews and AI Mode are built from the regular search index. The control for those is nosnippet or the Search Console generative AI setting.

Does opting out of AI training remove me from Gemini answers?

It can. Google says it will treat a training opt-out as a Google-Extended disallow unless you name Google-Extended yourself. Google-Extended covers grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, and Google says sites that allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

Can I block Gemini training but stay in Gemini grounding?

Not with Google's current controls. Google-Extended is one token covering both training of future Gemini models and grounding at prompt time. Allowing it allows both, disallowing it refuses both. If that changes, it will show up in Google's crawler documentation first.

Does the new token affect ChatGPT search?

OpenAI's own summary, published 30 October 2026, says it will honor a shared robots.txt signal for AI training preferences. Nothing it has published says the signal touches search. OpenAI's search crawler is OAI-SearchBot, a separate token from its training crawler GPTBot, and you control it separately.

Does my existing User-agent: * block count as a training opt-out?

No. The IETF draft says wildcard groups must not be read as matching the policy token. If your file has no User-Agent: __usage_ai-training line, you've expressed no preference under the shared token. Your wildcard rules still apply to crawlers and to Google-Extended in the ordinary robots.txt way, so a User-agent: * Disallow can already keep pages out of Gemini.

Sources and further reading

Related reading

AI crawlers explained: who reads your site
Do llms.txt and robots.txt matter for AI search?
Should you opt out of Google AI Overviews?

Arnav Mukherjee

Founder, TofuBofu