Should you add the shared AI training policy token to robots.txt?

Not yet in full, but yes in part. Add the new __usage_ai-training line to your robots.txt alongside your existing GPTBot, Google-Extended and Bing rules, because it costs nothing and signals intent. Don't delete anything else: the companies that announced support for it are rolling it out on wildly different timelines, and Google's own crawler documentation hadn't even switched over as of early November 2026.

What the new policy token actually is

On 30 October 2026, Google announced plans to support a shared AI training control in robots.txt, framing it as a way for site owners to express AI training preferences across multiple providers in one step, building on work from the AIPREF working group at the IETF (Google's Search Central blog). In Google's own words, "Instead of managing separate instructions for every AI developer, website owners will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers" (Google's Search Central blog).

The underlying spec is an individual IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, authored by Kevin Kelley of Anthropic and dated 30 October 2026, submitted to the IETF AI Preferences working group (Dev.to). It has a defined shelf life: the draft expires 3 May 2027 unless renewed or replaced, according to Emergent's reporting (Emergent).

How the __usage_ai-training syntax works

The draft's design is deliberately minimal. It introduces no new directives and no new grammar, reusing RFC 9309, the existing robots.txt standard, unchanged, with the same path language, longest-match precedence and group structure that crawlers already understand (Dev.to).

AI Weekly reports that OpenAI, Google, Microsoft and Anthropic all said on 30 October that they will honour a single robots.txt policy token, __usage_ai-training, letting a site opt its content out of AI model training across every one of their crawlers in one line (AI Weekly). The draft also closes a loophole that worried publishers: a User-Agent: * group never matches the policy token, and a robots.txt file without a dedicated __usage_ai-training group expresses no training preference at all (Robotstxt.com). Your existing wildcard block does nothing here, deliberately.

Who has committed, and when each will actually honour it

The headline announcement names four companies, but the fine print matters more. Implementation timelines vary sharply: Anthropic has already begun implementing it, OpenAI says it will follow in the coming months, Microsoft is targeting 1 February 2027, and Google is targeting 2027 generally (AI Weekly). As of 5 November 2026, Google's own crawler documentation still listed Google-Extended as the working AI-training control, with only a note that Google intends to support the shared token in early 2027 (Digital Applied).

Coywolf's analysis found all four companies are keeping their old proprietary tokens running in parallel while the new shared token doesn't yet work for all of them (Coywolf). Meta, Apple and Perplexity were not part of the 30 October announcements, so the token currently covers a subset of the AI ecosystem, not all of it (Cellcog).

Perplexity is worth understanding on its own terms. Its help centre states that PerplexityBot indexes pages similarly to other search engines, and because Perplexity does not build foundation models, content fetched this way isn't used for AI model pre-training (Perplexity's help centre). Separately, Perplexity-User fetches a page only when a person asks a question, and Momentic Marketing reports this fetcher "generally ignores robots.txt rules" since a user requested it (Momentic Marketing). If you're weighing how much attention to give Perplexity specifically, read our take on whether you should stop tracking Perplexity in your AI visibility reports.

What it does and doesn't cover

The token's definition of training is narrow and explicit: "AI Training" means using an asset to modify the learned parameters of a generative AI model, excluding models used for exclusively non-generative tasks (Dev.to). The scope stops at training data collection, leaving search indexing and real-time retrieval under separate rules entirely (SaaS Sentinel). Retrieval-time use at OpenAI and Anthropic also falls outside the token (Shattered.io).

This distinction matters if you're thinking about whether ChatGPT or Gemini cite you in answers, as opposed to whether they train on you. If your real concern is citation behaviour rather than training data, this token isn't the lever to pull. You'd want to look at how models decide what to cite, which is the subject of our piece on how GPT-6 Astra changes ChatGPT's source-citing decisions.

Why Google's version is tied to Gemini and search visibility

Google's implementation carries a trade-off the others don't. Google states that when a site uses the policy token to opt out of training without explicitly naming Google-Extended, Google plans to interpret that as if the publisher had also disallowed Google-Extended (Google's Search Central blog). PPC Land's reporting, as summarised by Tech Insider, says this means Google treats the shared token as a block on both training and Gemini grounding (Tech Insider).

For SEO-focused publishers this is a real cost, not a theoretical one. PalAI Media frames it directly: sites that opt out of training via the universal token will automatically lose direct clicks and citations in Google's Gemini ecosystem responses (PalAI Media). Given how much search volume now surfaces through AI-generated answers, as we've covered in our analysis of AI Overviews and buying-intent search queries, that's not a small trade to make casually. If you want Gemini visibility but not training, name Google-Extended explicitly rather than relying on the shared token alone.

Is it a real standard, and can it be enforced?

No. No working group has approved the draft, and it currently has no standard status, according to Pasquale Pillitteri's analysis (Pasquale Pillitteri). Emergent's summary underscores the voluntary, non-binding nature of the arrangement: the move represents the first coordinated attempt by major Western AI labs to standardise publisher preferences for training data, though timelines vary and the mechanism carries no enforcement power (Emergent).

Google frames this as groundwork for something bigger, saying support across major AI providers "will enable a future standard that is cross-industry, robust, and durable," and ties the move to pressure from publishers, civil society groups and regulators (Google's Search Central blog). Google also notes the proposal is consistent with the EU Code of Practice for GPAI Model Providers under the EU AI Act, which already recognises robots.txt as a primary method for publishers to express machine-readable preferences (Google's Search Central blog).

Don't confuse this with Content Signals (ai-train=no), a separate, overlapping mechanism. Google does not read it, Google's robots.txt spec supports only four fields and ignores the rest, and OpenAI, Anthropic and Perplexity don't mention it in their crawler documentation either (Contextbolt).

Our view: this is a sensible, low-cost step forward, but calling it "the new standard" today is premature. An unratified draft with a May 2027 expiry date and four different go-live dates is a signal of direction, not a settled protocol you should restructure your crawler strategy around. The real limitation is that enforcement is entirely voluntary: nothing in the spec stops a crawler from ignoring the token, and the evidence that it changes real-world training behaviour is thin because most participants haven't implemented it yet.

What to put in your robots.txt today

How2Shout's analysis of the announcement recommends adding the shared token now, but leaving your existing GPTBot, Google-Extended and Bing rules exactly where they are until each company confirms its support is actually live (How2Shout). That's the right move. Add the __usage_ai-training group, keep your proprietary tokens running in parallel as Coywolf's analysis shows the big four are themselves doing, and revisit the file once Google's documentation actually reflects the change rather than just promising it.

If you're also weighing broader crawler-control questions, our piece on whether you should enrol in Cloudflare's Pay Per Crawl covers a related but separate mechanism worth understanding before you touch robots.txt again.

Frequently asked questions

Does adding the shared token replace Google-Extended, GPTBot and other existing rules?

No. How2Shout's analysis of the announcement recommends keeping existing GPTBot, Google-Extended and Bing rules in place and adding the shared token alongside them, since support isn't uniformly live yet. All four companies announcing support are themselves running old and new tokens in parallel.

Will opting out of AI training also stop my site appearing in Google's AI Overviews?

Potentially, if you use Google's implementation without naming Google-Extended separately. Google says it will treat an opt-out via the shared token as blocking Google-Extended by default, which PPC Land's reporting says extends to Gemini grounding too.

Does this token stop Perplexity from using my content?

Not in the way it does for the other four companies. Perplexity says it doesn't build foundation models, so pages crawled by PerplexityBot aren't used for pre-training regardless, and its user-triggered fetcher generally ignores robots.txt entirely.

Is the __usage_ai-training token the same thing as Content Signals (ai-train=no)?

No, they're separate and overlapping mechanisms. Google doesn't read Content Signals, its robots.txt parser only supports four fields, and OpenAI, Anthropic and Perplexity don't mention it in their documentation either.