We Read 53 Devtool robots.txt Files: None Use the New AI Training Token

A robots.txt AI training opt-out now has one shared token: __usage_ai-training. We read the robots.txt files of 53 developer tool sites on November 1, 2026. Not one uses the token yet, and 62% have no AI rule of any kind.

What is the __usage_ai-training token?

The __usage_ai-training token is a robots.txt group that states whether your content may be used to train AI models. It does not control crawling. It comes from an Internet Engineering Task Force (IETF) Internet-Draft dated October 30, 2026, written by Kevin Kelley of Anthropic. [1]

The draft adds no new directives. It reuses the User-Agent line as a policy token, with the normal path matching rules. A site that wants no training except on its press kit writes:

User-Agent: __usage_ai-trainingDisallow: /Allow: /press-kit/

Disallow means a preference against training on that path. Allow means a preference in favor. No group means no preference at all. [1] The draft is an individual submission and has no formal standing in the IETF standards process. [1]

Which AI companies support the training token, and when?

Bing, Google and OpenAI announced support on October 30, 2026. Only Bing has published a firm date. Anthropic wrote the draft. Sources: Bing [2], Google [3], OpenAI [4], IETF draft [1].

Bing adds a second date. On March 1, 2027, NoArchive and NoCache stop counting as training or grounding preferences, so sites that rely on them need to switch. [2] Bing also says the training setting does not change eligibility for Bing results. [2]

Google adds one rule worth reading twice. If you name Google-Extended yourself, that group wins over the shared token for Google. [3]

What this means: until 2027, the token is a statement of intent. The crawler-specific tokens below are the controls that work today.

What did we find on 53 developer tool sites?

Almost nobody has acted. No site uses the new token, and most sites have no AI policy written in robots.txt at all. Counts are from the EarShots scan on November 1, 2026.

The 2 sites that say no to training are twilio.com and heroku.com. Both say it only through Content-Signal. Neither names a training crawler. The 2 Bytespider blocks are github.com and postman.com.

The sites that name AI crawlers mostly welcome them. Netlify, Zapier and Retool give GPTBot and ClaudeBot an explicit Allow: /. Postman's file explains why in a comment: "Model-training crawlers, so models learn Postman by default."

What this means: devtool companies want to be in the models. Developers ask models how to use an application programming interface (API), and a model that trained on your docs is more likely to answer with your syntax. That last point is a reasonable inference, not something any platform documents.

Does User-agent: * block AI training?

No. Under the draft, a wildcard group never matches the training token. [1] A site that blocks everything with User-agent: * has stated no training preference.

The rule runs the other way too. A permissive wildcard group does not opt you in. If you want models to train on your docs, the draft only counts an explicit group:

User-Agent: __usage_ai-trainingAllow: /

Does blocking AI training hurt AI search visibility?

Mostly no, with one exception at Google. Each major AI company documents training and search as separate tokens.

OpenAI describes the split directly: a site can allow OAI-SearchBot to appear in search results and disallow GPTBot to keep content out of training. [5] Anthropic warns that blocking Claude-User or Claude-SearchBot may reduce visibility in Claude's search. [6]

Google is the exception. Google says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. [3] A shared training opt-out with no Google-Extended group is read as a Google-Extended disallow, so it opts you out of that grounding too.

Bing handles grounding as a separate, page-level preference, not through robots.txt. [2]

What we cannot tell you: whether training data changes how often a model names your product without search. No platform documents that, and we have not measured it.

What is the catch with naming an AI crawler?

A named group replaces your wildcard rules for that crawler. Under RFC 9309, a crawler obeys only the most specific group that matches its user agent. [8] A bot named with Allow: / ignores every Disallow you wrote under User-agent: *.

Example: your wildcard group blocks /internal-search/. You add a GPTBot group with Allow: /. GPTBot may now fetch /internal-search/. The fix is to repeat the path rule inside the named group:

User-agent: *Disallow: /internal-search/ User-agent: GPTBotAllow: /Disallow: /internal-search/

Render and Postman already do this by repeating their path rules inside each AI group.

What should a devtool robots.txt say now?

Pick a training policy, state it in the new token, and keep the crawler tokens that work today. This block is for a team that wants AI search visibility but no training:

# Shared training preference (Bing from Feb 1, 2027; Google and OpenAI later)User-Agent: __usage_ai-trainingDisallow: / # Training crawlers that read their own tokens todayUser-agent: GPTBotDisallow: / User-agent: ClaudeBotDisallow: / # Optional: allow this to stay eligible for Gemini Apps groundingUser-agent: Google-ExtendedAllow: /

Leave OAI-SearchBot, Claude-SearchBot, Bingbot and Googlebot under your wildcard rules so they stay eligible for search. If you also want out of Gemini Apps grounding, change the Google-Extended group to Disallow: /, or delete it and let the shared token apply. A team that wants training flips each Disallow: / to Allow: /.

Four steps, in order:

1. Decide the policy with docs, legal and developer relations in one meeting.
2. Add the __usage_ai-training group. Stating it early costs nothing.
3. Repeat any wildcard Disallow paths inside each named AI group.
4. Check AI referrals and crawl requests monthly.

For crawler-by-crawler detail, read ChatGPT Search crawler access: a publisher's guide and Claude web crawlers: a site owner's guide. For the traffic side, read AI crawler access and referral traffic for documentation. For Google's position on special AI files, read Does Google require special AEO or GEO optimization?

To see whether a policy change moves your answer engine optimization (AEO) visibility in ChatGPT, Gemini, AI Overviews and Claude, track it in EarShots AI Search.

Methodology

We chose 60 developer tool and open source project domains by hand. The list covers hosting, databases, frameworks, languages, observability, auth and AI tooling. We fetched https://<domain>/robots.txt on November 1, 2026 and read every group.

• Readable: 53 of 60. Six failed with fetch errors: nextjs.org, rust-lang.org, npmjs.com, tailwindcss.com, flutter.dev and stackoverflow.com. segment.com redirected to twilio.com, so we excluded it.
• Redirects: neon.tech redirected to neon.com. We counted the neon.com file.
• Classification: "Names an AI crawler" means a group for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot or CCBot. "Blocks" means Disallow: / in that group.

Limits. The sample is hand-picked, so it is not random. We read root domains only, and docs subdomains may differ. robots.txt states a preference. It does not prove what any crawler does. This is a snapshot, and we will re-run it after Bing's February 1, 2027 date.

Sources

1. Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00). IETF, October 30, 2026.
2. Your content, your choice: clearer preferences for training and grounding. Microsoft Bing, October 30, 2026.
3. Supporting a shared AI training control for websites. Google Search Central, October 30, 2026.
4. Making it easier for websites to express AI training preferences. OpenAI, October 30, 2026.
5. Overview of OpenAI crawlers. OpenAI.
6. Does Anthropic crawl data from the web, and how can site owners block the crawler?. Anthropic.
7. Google's common crawlers. Google for Developers.
8. RFC 9309: Robots Exclusion Protocol. IETF.

Last verified November 1, 2026. Next refresh: February 2, 2027.