How to Block AI Crawlers in robots.txt (New AI Training Policy Token Guide)

Blocking AI crawlers used to mean adding a new robots.txt rule for every AI company, one bot at a time.

On October 30, 2026, Anthropic, OpenAI, Google and Microsoft backed a single shared robots.txt “policy token”, __usage_ai-training, that lets you tell all of them at once whether your content can be used to train AI models.

This guide explains the new token, gives you every major AI user agent with its current name, and shows you ready-to-copy robots.txt files, WordPress and Cloudflare options, and how to test that it all works.

What is the new AI training policy token?

robots.txt has always worked by naming a crawler in a User-Agent line and then listing what it may or may not fetch. The problem with AI is that every company has its own crawler names, and new ones keep appearing.

A site that wanted to opt out of AI training had to track GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and many more.

The new approach adds a policy token. Instead of naming a crawler, the User-Agent line names a use: __usage_ai-training. Any AI company that supports it reads that group as your preference about using your pages to train generative AI models.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example, taken from the spec, says “do not use my site for AI training, except the /press-kit/ folder.”

The token is defined in an IETF Internet-Draft called “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal-00), written by Kevin Kelley of Anthropic and dated October 30, 2026.

It is an individual draft, so it is not yet an official IETF standard, and it expires on May 3, 2027 unless it is updated. It builds on the vocabulary of the IETF AI Preferences (AIPREF) working group.

Who supports it and when

The Verge reported that the four biggest AI players are on board, with different start dates. Here is the timeline based on each company’s own statements.

Two details from Google are worth knowing. First, if you use the shared token to opt out and you do not mention Google-Extended, Google says it will treat that as if you had disallowed Google-Extended.

Second, sites that specifically allow Google-Extended stay eligible to appear in Gemini Apps answers through grounding. Meta’s communications head has publicly backed a common signal but has not said Meta will honor this token.

How the token works (rules from the spec)

The draft keeps the normal robots.txt format from RFC 9309. These are the rules that matter in practice:

• Exact match, any case. Consumers must match __usage_ai-training exactly, ignoring upper or lower case.
• Longest match wins. Path lookups follow the normal robots.txt longest-match rules. Disallow means “please do not train on this”, Allow means “training is fine”.
• No group, no preference. If your file has no group with this token, you have expressed no preference either way.
• The wildcard does not count. A User-Agent: * group must not be read as a training preference. So Disallow: / under * blocks crawling but says nothing under this spec about training.
• It is not a crawl rule. The token does not stop any bot from fetching your pages. To limit crawling you still name the crawler itself.
• Specific beats general. Companies may treat their own tokens (like GPTBot or ClaudeBot) as more specific than the shared token.
• You can combine tokens. A group can list several User-Agent lines, mixing policy tokens and crawler names.

“Training” here has a narrow meaning. The AIPREF vocabulary defines it as using content to change the learned parameters of a generative AI model. Models used only for classification, ranking or scoring are outside it.

So this token is not about search ranking, and it is not about AI tools fetching your page live to answer a question.

Note: It is also only a request. robots.txt is voluntary, the draft says private agreements can override it, and it does not remove your content from models that were already trained.

Every AI crawler user agent you should know

This list is based on each company’s own documentation, where one exists. The “Type” column is the key to the trade-off in the next section.

Notice the pattern. Most big companies now split their bots into three jobs: training, search and user-triggered fetches. That split is what lets you block one without the others.

The big trade-off: training vs AI search visibility

Before you paste a “block everything” file, think about what you want.

If you block training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent), your future content should stay out of new training runs. You lose nothing in classic search. Google says Google-Extended does not affect inclusion or ranking in Google Search, and Apple says blocking Applebot-Extended does not remove you from Spotlight, Siri or Safari results.

If you also block search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer), you drop out of AI answers that cite and link sources. OpenAI says opted-out sites are left out of ChatGPT search answers, apart from navigational links. That matters more each month.

My look at ChatGPT search reaching 20 percent of Google shows how big this traffic source is becoming. If you are not sure which assistants your readers use, my ChatGPT vs Claude vs Gemini vs Grok comparison covers the main ones.

User-triggered bots (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) act for a person in real time, so the companies say robots.txt may not apply. To truly stop them, you need a firewall rule.

Google is a special case. AI Overviews and AI Mode use content crawled by normal Googlebot. Blocking Google-Extended does not remove you from them. To limit how your text appears there, Google points to snippet controls such as nosnippet, data-nosnippet and max-snippet, which also affect normal search snippets.

For most blogs and businesses, the sweet spot is: block training, allow search. You keep getting cited and clicked, without feeding new training runs.

robots.txt examples you can copy

Your robots.txt must sit at the root of each host, for example https://example.com/robots.txt. Each subdomain needs its own file.

1. Block AI training, allow AI search (recommended for most sites)

# Shared AI training preference (new policy token)
User-Agent: __usage_ai-training
Disallow: /

# Company-specific training controls (keep until each company confirms support)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# AI search bots: allowed so you can be cited
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
Allow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

2. Block all AI training and AI search

User-Agent: __usage_ai-training
Disallow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /

User-agent: *
Allow: /

3. Allow training on some folders only

User-Agent: __usage_ai-training
Disallow: /
Allow: /docs/
Allow: /press/

4. Explicitly allow everything for AI (maximum AI reach)

User-Agent: __usage_ai-training
Allow: /

User-agent: *
Allow: /

A few notes. Listing several User-agent lines above one rule set is valid robots.txt. Group lines are case-insensitive. And do not block Googlebot or Bingbot unless you want to leave normal search too.

Microsoft’s page-level training and grounding tags

Microsoft went further than the others. From February 1, 2027, it will honor the shared token in robots.txt and also two new page-level preferences for Bing, Copilot and Microsoft 365.

<!-- Opt this page out of AI training -->
<meta name="Bingbot" content="ai-training:disallow">

<!-- Block use of this page in AI answers (grounding) -->
<meta name="Bingbot" content="ai-grounding:disallow">

<!-- Allow Microsoft's own AI answers, block third-party grounding APIs -->
<meta name="Bingbot" content="ai-grounding:disallowextended">

The same values work as HTTP headers, for example X-Robots-Tag: Bingbot: ai-training:disallow. Grounding can only be set per page, not in robots.txt. DisallowExtended keeps your page in Microsoft-run experiences such as Copilot and Microsoft 365 but excludes grounding services sold to third parties, like Web IQ and the Foundry grounding API.

Warning: If you used NoArchive or NoCache to keep content out of Bing Chat or Copilot, pay attention. After March 1, 2027, Microsoft stops reading those tags as training or grounding preferences. A temporary grounding setting opens in Bing Webmaster Tools on December 15, 2026.

Record it by March 1, 2027 and publish the new tags, or your pages default to Allow for grounding until recrawled. Microsoft also says these preferences do not affect normal Bing search results.

How to edit robots.txt on WordPress

WordPress creates a virtual robots.txt if no real file exists. You have three easy ways to change it.

1. With Yoast SEO: go to Yoast SEO > Tools > File editor, create or edit robots.txt, paste your rules and save.
2. With Rank Math: go to Rank Math SEO > General Settings > Edit robots.txt, paste your rules and save.
3. By hand: create a plain text file named robots.txt, paste your rules, and upload it to your site’s root folder (the same folder as wp-config.php) with your host’s file manager or SFTP. A real file replaces the virtual one.
4. Clear your page cache and CDN cache so the new file is served.
5. Open https://yoursite.com/robots.txt in a private window and check that your new lines are there.

Warning: Do not use Settings > Reading > Discourage search engines for this. That setting asks all search engines to stay away and can remove you from Google.

Cloudflare options: managed robots.txt and AI Crawl Control

If your site runs behind Cloudflare, you get two extra tools, both available on all plans according to Cloudflare’s docs.

Managed robots.txt (Bot Preference Sync) writes AI crawler rules into your robots.txt for you, based on your Search, Agent and Training policies. If you already have a robots.txt, Cloudflare adds its block in front of yours. To set it up:

1. In the Cloudflare dashboard, open your domain and go to Security Settings.
2. Filter by Bot traffic.
3. Open AI Crawler and Agent access policies.
4. Set Training to Disallow (and choose your Search and Agent policies).
5. Turn on Enable Bot Preference Sync and click Save.

Cloudflare’s managed block disallows bots such as Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. It also adds Cloudflare’s own Content Signals line, which by default reads search=yes, ai-train=no.

Setting Training to Disallow also turns on edge blocking for AI training crawlers, except mixed-use crawlers. Google Search Console may show “Syntax not understood” for the content signals line; Cloudflare says it has seen no effect on crawling or SEO.

AI Crawl Control shows which AI services visit your site and how often, lets you allow or block each crawler, tracks which ones obey your robots.txt, and offers pay per crawl in private beta. This is the tool to use when a bot ignores robots.txt, because it blocks at the network edge.

Not on Cloudflare? Most hosts and firewalls (and plugins like Wordfence) let you block by user agent. For bots that fake their name, match the published IP lists too: OpenAI, Anthropic, Perplexity and Common Crawl all publish JSON lists of their crawler IPs.

How to test your setup

1. Open https://yoursite.com/robots.txt in a browser. Check that the file loads with status 200 and shows your new rules.
2. Fetch it from the command line to bypass browser cache: curl -s https://yoursite.com/robots.txt.
3. In Google Search Console, open Settings > robots.txt to see the version Google fetched and any parse errors.
4. Test a page as a specific bot, for example: curl -I -A "GPTBot" https://yoursite.com/some-page/. A 403 means your firewall blocks it; a 200 means only robots.txt (a request) is in play.
5. Check your server or CDN logs for AI user agents over the next few days. Remember many companies cache robots.txt for up to about 24 hours.
6. Verify real bots by IP. Compare the request IP with the company’s published list, or use reverse DNS for Common Crawl (hostnames end in crawl.commoncrawl.org).

# Quick log check for AI bots (nginx/apache access log)
grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|CCBot|Bytespider|meta-externalagent|Amazonbot|Applebot" /var/log/nginx/access.log | awk '{print $1, $12, $13}' | sort | uniq -c | sort -rn | head

There is no way to “test” the __usage_ai-training token itself today, because only Anthropic reads it so far. Just confirm the group is in your file and spelled exactly right, with two underscores at the start.

My Take

This is the most useful robots.txt change in years. One line that covers every major AI lab is far better than chasing a new bot name every month. The fact that Anthropic, OpenAI, Google and Microsoft all signed up gives it a real chance of becoming the default.

But do not delete your old rules yet. Only Anthropic reads the token today, the others start over the next few months, and it is still an individual draft.

Add the token now, keep your GPTBot, ClaudeBot and Google-Extended lines, and think hard before you block AI search bots. Being cited in ChatGPT, Claude and Perplexity answers is becoming real traffic, and blocking training does not require giving that up.

The labs behind the top models on my November 2026 AI model leaderboard are the same ones that run these crawlers, so this choice touches every major assistant.

Troubleshooting: Common Problems and Fixes

My robots.txt changes do not show up

Clear your WordPress cache plugin and CDN cache. If you use Yoast or Rank Math and also uploaded a real file, the real file wins. Check the live URL in a private window.

AI bots still visit after I blocked them

Crawlers can cache robots.txt for up to about 24 hours. User-triggered bots like ChatGPT-User and Perplexity-User may not follow robots.txt at all. Use a firewall or Cloudflare AI Crawl Control to enforce the block.

I blocked Google-Extended and still appear in AI Overviews

That is expected. AI Overviews use normal Googlebot crawling. Use snippet controls like nosnippet or max-snippet if you need to limit it, knowing they also change your normal search snippets.

My traffic from ChatGPT or Perplexity dropped

You probably blocked OAI-SearchBot or PerplexityBot. Allow those search bots and keep training bots blocked.

Search Console shows “Syntax not understood”

This is usually Cloudflare’s Content-Signal line or a typo. Cloudflare says the content signals line has no effect on crawling. For typos, check the spelling of every User-agent line and use one directive per line.

Blocking by IP stopped the bot from reading robots.txt

Anthropic warns about this. If a bot cannot read robots.txt, it cannot see your opt-out. Prefer robots.txt plus user-agent rules, and use IP lists to verify traffic.

Frequently Asked Questions

What is the __usage_ai-training token?

It is a robots.txt policy token from an IETF Internet-Draft by Anthropic. You put it in a User-Agent line, and supporting AI companies read the Disallow and Allow rules under it as your AI training preference.

Does the token stop AI bots from crawling my site?

No. It only states a preference about training. To stop crawling, name the crawler itself, and use a firewall for bots that ignore robots.txt.

Which companies honor it right now?

Only Anthropic so far. OpenAI says in the coming months, Microsoft from February 1, 2027 and Google in early 2027.

Will blocking AI training hurt my Google rankings?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. Blocking Googlebot would, so leave it allowed.

Can I block training but still appear in ChatGPT search?

Yes. Block GPTBot and allow OAI-SearchBot. OpenAI treats them separately.

Does it remove my content from models already trained?

No. It only affects future use. Content already used in training is not removed by a robots.txt change.

Do I still need GPTBot and ClaudeBot rules?

Yes, for now. The companies say their existing controls keep working, and they are your only option until each one supports the shared token.

Sources

• The Verge: OpenAI, Google and Microsoft back an AI scraping opt-out standard
• IETF Datatracker: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)
• Bing Blog: Clearer preferences for training and grounding
• Google Search Central: Google common crawlers (Google-Extended)
• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Apple: About Applebot
• Perplexity: Perplexity crawlers
• Amazon: About Amazonbot
• Common Crawl: CCBot
• Meta: Meta web crawlers
• Cloudflare Docs: Managed robots.txt
• Search Engine Roundtable: Google to support shared AI training control
• Coywolf: Google, OpenAI, Microsoft and Anthropic back a shared robots.txt AI training signal