Module 4 min
AI Crawlers
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended are distinct agents with distinct rules; block the wrong one and you vanish from an engine.
Why it mattersA single robots.txt line can delete you from an entire answer engine.
Definition & Foundation
What it is, in plain words
AI crawlers are not one bot. Each vendor runs a small fleet split by job: a training bot collecting content for future models, a search-index bot feeding the engine's live answers, and a user bot fetching a page on demand when someone asks. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended are distinct agents with distinct rules, and because access is binary, a single robots.txt line can delete you from an entire answer engine.
The allow/block decision is a real trade-off, not a formality. Cloudflare's mid-2025 data put the crawl-to-referral imbalance starkly: roughly 14:1 for Google, 1,700:1 for OpenAI, and Anthropic's crawler near 71,000 pages fetched per referral sent; the old crawl-for-traffic bargain has broken down for training bots. But blocking the retrieval bots to punish that imbalance removes you from the answers your buyers read. The skill this node teaches: know which bot does which job, and make each allow/deny an explicit, revisited decision.
The Crawler Fleet Map
Who fetches for which product, and the decision each one carries
OpenAI: GPTBot · OAI-SearchBot · ChatGPT-User
GPTBot collects training data for future models. OAI-SearchBot builds the ChatGPT Search index. ChatGPT-User fetches live when a user asks. Block GPTBot and you've opted out of training only; block the other two and you vanish from ChatGPT's answers.
Anthropic: ClaudeBot · Claude-SearchBot · Claude-User
ClaudeBot feeds model training; Claude-SearchBot indexes for Claude's search; Claude-User fetches on user request. Each is independently controllable in robots.txt; you can stay visible in Claude while staying out of training.
Perplexity: PerplexityBot · Perplexity-User
PerplexityBot builds the index behind Perplexity's cited answers; Perplexity-User fetches pages users ask about. Perplexity is an answer engine, not a model vendor; blocking these mostly just removes your citations.
Google: Googlebot vs Google-Extended
Google-Extended is a robots.txt token governing use of your content for Gemini training and grounding; it is not the crawler behind AI Overviews. AI Overviews and AI Mode run on the normal Search index via Googlebot; you can't leave them without leaving Search itself.
Microsoft: Bingbot
One crawler feeds Bing, Copilot, and (via Bing's index) parts of ChatGPT's retrieval. Blocking Bingbot quietly removes you from multiple AI surfaces at once; verify in Bing Webmaster Tools and treat it as essential.
The long tail
CCBot (Common Crawl; feeds many research datasets), Applebot-Extended (Apple AI training opt-out), Meta-ExternalAgent (Meta training), plus new agents appearing every quarter. Default policy beats ad-hoc: decide once per category (training vs retrieval), review quarterly.
Myths vs Reality
Common misreadings, corrected
nosnippet), with the trade-offs those carry.Implementation Guide
Explicit policy, verified access, monitored logs
This is a genuinely technical node: the work is a robots.txt policy you've consciously chosen, verification that the bots you want actually get 200s, and a log habit that catches surprises.
The crawler-policy playbook
Inventory who crawls you today
Pull 30 days of server or CDN logs and count hits by user-agent: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bingbot, Google-Extended token checks. No AI hits at all usually means an upstream block (WAF, bot protection) you didn't know about.
Decide policy per category, in writing
Training bots (GPTBot, ClaudeBot, CCBot, Applebot-Extended, Meta-ExternalAgent): a business/licensing decision; many publishers block, many brands allow for parametric familiarity. Retrieval + user bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Bingbot): allow unless you have a specific reason not to; they are your GEO visibility.
Write the robots.txt explicitly
Name each bot rather than relying on wildcards, e.g. User-agent: GPTBot → Disallow: / next to User-agent: OAI-SearchBot → Allow: /. Explicit rules survive team turnover and make the policy auditable at a glance.
Verify access end to end
robots.txt is only a request; confirm reality: fetch key pages with each bot's user-agent string, check for 200s and full HTML, and confirm your CDN/bot-management layer (Cloudflare et al.) isn't challenging or blocking agents your robots.txt allows. Cloudflare has blocked many AI crawlers by default for new sites since mid-2025.
Monitor monthly; review policy quarterly
New agents appear constantly, and vendors add bots (or change behavior) without fanfare. A monthly log glance plus a quarterly policy review keeps the decision current, and catches the day an engine stops (or starts) fetching you.
On your platform
User-agent: GPTBot / Disallow: /, while leaving OAI-SearchBot and PerplexityBot allowed.robots.txt.liquid (Online Store → Themes → Edit code → Add template); Shopify serves a default you can extend, not replace blindly./robots.txt directly and confirm the rendered rules match your intent.app/robots.ts so it's code-reviewed like everything else: return per-bot rules arrays for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot.request.headers.get('user-agent')) to a analytics sink; this is your retrieval telemetry.public/robots.txt with the explicit per-bot policy; it deploys with every build and diffs cleanly in git.curl -A "GPTBot" (and each other UA) that responses match the policy.grep -c per bot is enough telemetry to start.Verification Checks
How to know it's really done
Your crawler policy is real when the logs confirm it. Click a check to mark it verified:
Update Cadence & Dependencies
Keeping it alive
Works together with
Impact Weightage & Results TAT
What it moves, and how fast
A gate, not a gradient: retrieval access is binary, and one wrong line (or one WAF preset) zeroes out every other GEO investment on the affected engine.
Mostly orthogonal to classic SEO, but Bingbot hygiene and rendering checks pay into both games.
Editorial estimate of this node's contribution to your total GEO / SEO outcome. Nodes overlap, so weights don't sum to 100.
Tools for this
Go deeper from How AI Engines Work
Voices to follow
- Michael King · iPullRank The deepest technical explanations of how AI retrieval and ranking actually work.
- Dan Petrovic · DEJAN Machine-learning-grounded research on embeddings, ranking and model behavior.
- Bartosz Góralewicz · ZipTie Hands-on reverse-engineering of how AI Overviews and Perplexity pick sources.
Always current
These links resolve live: what you get is generated or filtered the moment you click.