- Every major AI company now runs separate bots for separate jobs: one to train models, one to index for search, one to fetch a page live when a user asks.
- GPTBot is not OAI-SearchBot. ClaudeBot is not Claude-SearchBot. Blocking the training crawler does not block the search crawler, so you can refuse training and still be cited.
- Blocking a search crawler is the expensive mistake: it removes you from that engine's answers completely, and your competitors fill the gap.
- Scope the training opt-out, never apply it site-wide. Your homepage, about page, and service pages are how a model learns your brand exists when it answers without searching. Protect the content you sell, not the pages that identify you.
- Google-Extended does not control AI Overviews. Those run on the regular Googlebot index, and Google says the token has no effect on Search inclusion or ranking.
- From September 15, 2026, new Cloudflare domains block Training and Agent bots on ad-displaying pages by default, while Search bots stay allowed.
- Robots.txt is only the first gate. WAF rules, bot management, and client-side rendering block far more AI crawlers than robots.txt ever does.
Somebody on your team read that AI companies were scraping the web for free, and switched on the block. Reasonable instinct. The problem is what it costs you now.
AI crawlers come in three jobs, and only one of them is training. A training bot collects content to build a model. A search bot indexes your pages so the engine can cite you later. An agent fetches your page live because a person just asked a question. Block training if you want. Block search and you are invisible in AI answers.
Which AI crawlers should I actually block?
Block the training bots if you object to model training. Leave the search bots and the live-fetch agents alone, because those are the two paths that end with a user reading your name. Here is the current split, taken from each company's own crawler documentation.
| Company | Training bot | Search bot (earns citations) | Live user fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | No separate public training bot | PerplexityBot | Perplexity-User |
| Google-Extended (a control token, not a crawler) | Googlebot | Googlebot |
OpenAI also runs OAI-AdsBot, which checks the safety of pages submitted as ads in ChatGPT. It is not part of the citation path, so it rarely belongs in this decision.
Read the table one row at a time and the rule falls out on its own: the left column is optional, the middle column is your visibility, and the right column is a real person waiting on an answer. The chapter on AI crawlers in our playbook keeps the full list current as vendors add bots.
I blocked GPTBot. Did I just disappear from ChatGPT?
No, and this is the single most common misunderstanding in the whole topic. GPTBot crawls content that may be used to train OpenAI's foundation models. ChatGPT's search feature is served by a different crawler, OAI-SearchBot, and when a user asks ChatGPT about a specific page, ChatGPT-User goes and fetches it. Disallowing GPTBot leaves both of those routes open.
Anthropic works the same way, and is unusually clear about it: ClaudeBot gathers training data, Claude-SearchBot indexes for search results, and Claude-User fetches a page when someone references it in a conversation. Three bots, three robots.txt tokens, three independent decisions.
So the sentence "we blocked AI" is meaningless until you know which token was used. It is the difference between opting out of training and opting out of the market. If you want the deeper mechanics of how each engine sources what it quotes, we cover them per engine in ChatGPT SEO and how to rank in Perplexity.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended is not a crawler at all. It is a control token you place in robots.txt to say whether your content may be used to train Gemini and ground certain generative products. The crawling itself is still done by Googlebot.
Because AI Overviews are generated from the same Search index that Googlebot builds, disallowing Google-Extended does not pull you out of them. Google has also said the choice has no effect on inclusion or ranking in Search. That makes Google-Extended the cleanest opt-out available anywhere in this list: you decline model training and lose no visibility for it.
What changes on September 15, 2026?
Cloudflare retired the old "block all AI bots" switch and replaced it with three independent controls that match the three jobs: Search, Agent, and Training. For each one you can block on all pages, block only on pages that display ads, or not block at all.
From September 15, 2026, domains newly onboarding to Cloudflare get new defaults. Training and Agent bots are blocked on pages that display ads. Search bots stay allowed. Cloudflare applies this to new customers, new sites added by existing customers, and all existing free customers, and says existing customers can opt out any time before that date.
One detail deserves attention if you sell anything through an assistant. The Agent category is the live fetch, the bot that runs when a person is waiting for an answer about you right now. Blocking Agent on your ad-supported pages is a defensible publisher decision and a costly one for a business trying to be recommended.
There is also a deliberate trap for mixed-use crawlers. Cloudflare says any setting that blocks Training also catches crawlers that combine Search and Training, which closes the loophole of collecting training data under the cover of search. Worth knowing before you assume Search-allowed means every search bot gets through.
My robots.txt is fine, so why do AI engines still miss me?
Because robots.txt is only the first of three gates, and it is the one that fails least often. The other two are quieter.
The first is your edge. A WAF rule, a bot-management setting, or a CDN configuration can reject an AI user agent before the request ever reaches your server. Nothing appears in robots.txt, nothing looks wrong in your CMS, and the engine simply never sees a page. This is the failure we find most often in audits, and it is usually switched on by someone protecting the site from scrapers.
The second is rendering. Most AI crawlers do not execute JavaScript. They read the HTML your server returns. If your content is assembled client-side, a crawler that is fully allowed still reads a near-empty document, and an empty document is never the source of a citation. The same applies to schema injected by a tag manager: it has to be in the static HTML to count.
Both of these are exactly what technical GEO exists to fix, and the playbook chapter on crawler access walks the full checklist.
What should my robots.txt actually say?
It depends on one decision: do you object to model training, or not? Both answers are legitimate, and both keep you citable, as long as you never touch the search bots.
Maximum AI visibility, training allowed
Every bot welcome. This is the right default for most businesses that want to be found and recommended, and it needs no AI-specific rules at all. Copy this, swap in your domain, and keep your own private paths on the Disallow lines.
# robots.txt: open to every crawler, including AI User-agent: * Allow: / # Your own private paths, not an AI decision Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml
Protect the content you sell, keep the pages that identify you
This is the version most brands actually want. It keeps the training crawlers out of the content you monetize, leaves your homepage, about page, and service pages in the corpus so models learn who you are, and explicitly allows every search and assistant bot. Change the two Disallow paths to whatever you are actually protecting.
# ---- Training crawlers: scoped, not site-wide ---- # Keep the pages that say who you are IN the training data. That is how # a model knows your brand exists when it answers without searching. # Disallow only what you protect commercially. Longest match wins, so # these Disallow lines beat the Allow above them. User-agent: GPTBot Allow: / Disallow: /research/ Disallow: /reports/ User-agent: ClaudeBot Allow: / Disallow: /research/ Disallow: /reports/ User-agent: Meta-ExternalAgent Allow: / Disallow: /research/ Disallow: /reports/ # Opt-out tokens. These make no requests of their own, they # tell an existing crawler not to use your content for training. User-agent: Google-Extended Allow: / Disallow: /research/ Disallow: /reports/ User-agent: Applebot-Extended Allow: / Disallow: /research/ Disallow: /reports/ # ---- Search and assistant bots: these earn your citations ---- User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # ---- Everyone else ---- User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml
One rule of robots.txt catches people out here. A crawler obeys only the block that names it, and it ignores the User-agent: * block entirely. So if your private paths matter, repeat those Disallow lines inside any named block that needs them. In the file above, the named search bots are deliberately given full access.
If you are a publisher licensing an archive, and the content itself is the product, swap those scoped Disallow lines for a single Disallow: / under each training agent. That is a legitimate commercial position. Just take it deliberately, knowing what the next section costs you.
What you should not do is copy a "block AI" snippet from a forum post written in 2024. Those lists predate the split, and they routinely disallow the search bots alongside the training ones. If you want a machine-readable pointer to your best content once crawlers are in, that is a separate file with a separate job: see whether llms.txt actually works, or generate one with our free llms.txt builder.
If I block training, will ChatGPT still know my company exists?
Less well, and this is where the blanket block quietly costs you. Search bots decide whether you can be cited when the engine goes and looks something up. Training decides what the model knows when it looks nothing up, which is what happens on broad questions like which company is best for a particular job. If your homepage, about page, and service pages were never in the corpus, there is no stored sense of your brand for the model to reach for.
So scope the opt-out to what you are actually protecting. Keeping original research, a licensed archive, or paywalled work out of a training set is a real commercial decision, because that content is the product. Withholding the pages whose entire job is telling the world who you are and what you sell buys you nothing and costs you recognition.
Be clear-eyed about the ceiling, though. Sitting in the training data does not win you "best in category." Those answers lean heavily on what independent sources say about you, which is earned rather than crawled, and it is why citation building is a separate discipline from crawler access. Training makes the model aware of you. Third parties are what make it recommend you.
How do I check which AI bots are hitting my site?
Read your server logs and filter by user agent. Your access logs are the only source that tells you what actually happened, as opposed to what your configuration intends. Look for the tokens in the table above, then answer two questions for each: did it get a 200, and did the response contain your real content.
A 403 or 429 against OAI-SearchBot, Claude-SearchBot, or PerplexityBot means an edge rule is quietly costing you citations. A 200 that returns an empty shell means rendering is. If you use Cloudflare, its bot analytics now break traffic out by the same three categories, so you can see Search, Agent, and Training separately rather than guessing.
Once access is clean, the question becomes whether the engines actually quote you, which is a different measurement entirely. That is what a GEO audit scores: mention rate, citation rate, and which competitor gets named in your place.
Frequently Asked Questions
Never block the search crawlers, because those are the ones that put you inside AI answers with a citation. Block training crawlers only on the content you monetize, such as original research or a licensed archive, and scope it by path rather than site-wide. The user agents are separate, so you can protect what you sell and still be quoted across ChatGPT, Claude, and Perplexity.
Yes, in almost every case. Training data is what a model draws on when it answers without searching the web, so it is how the model knows your brand exists and what you do at all. Excluding your homepage, about page, and service pages removes that knowledge and gains you nothing, since those pages are published to be read. Reserve the training opt-out for content that is itself the product.
No. GPTBot collects training data for OpenAI models. ChatGPT search results are powered by a separate crawler, OAI-SearchBot, and a third agent, ChatGPT-User, fetches pages when someone asks about your site in a chat. Blocking GPTBot leaves all three of those paths open.
No. Google-Extended is a robots.txt control token, not a crawler. It governs whether your content trains Gemini and grounds some generative products. AI Overviews are served from the regular Google Search index that Googlebot builds, so blocking Google-Extended does not remove you from them and Google says it has no effect on Search ranking or inclusion.
New domains onboarding to Cloudflare get updated defaults: bots classified as Training or Agent are blocked on pages that display ads, while Search bots stay allowed. The defaults apply to new customers, new sites added by existing customers, and all existing free customers. Existing customers can opt out before that date.
A training bot collects content to train or fine-tune a model. A search bot indexes your pages so the engine can answer questions about them later and cite you. An agent fetches a page live, in real time, because a person just asked something. Only the search and agent paths put you in front of a user today.
Mostly, with one important exception. Anthropic states all three of its bots honor robots.txt, including the user-triggered one. OpenAI and Perplexity draw a sharper line for user-initiated fetches: Perplexity documents that Perplexity-User can ignore robots.txt when a person supplies a specific URL as context, on the reasoning that the request came from a human, not a crawl.
Robots.txt is only the first gate. A WAF, bot-management rule, or CDN setting can block AI user agents before they reach your server, and most AI crawlers do not execute JavaScript, so client-rendered content reads as an empty page. Check your edge rules and your server-rendered HTML, not just robots.txt.
No. They do different jobs. Robots.txt controls access, which bots may fetch what. An llms.txt file is a proposed guide that points AI systems at your best content once they are already allowed in. It is optional and not a substitute for getting crawler access right.
The Bottom Line
The instinct to block AI companies was never wrong. The execution was. Every major vendor now separates the crawler that trains a model from the crawler that earns you a citation, which means you can decline the first without surrendering the second. Blocking by company is a blunt instrument. Blocking by job is the decision worth making.
Check three things this week: your robots.txt tokens, your edge rules, and whether a crawler with JavaScript disabled can still read your page. Most sites we audit fail on the second or third, never the first. Want the whole system, from crawler access through to citation tracking? Work through the GEO Playbook.