Module 6 min

Crawler Access

robots.txt directives are per-user-agent; your allow/deny choices decide which engines may ingest you at all.

By Akash Sharma SEO, AEO & GEO Strategist · Founder, The First Ranker

Why it mattersBlocking a training bot to protect IP can also remove you from that engine's answers.

Definition & Foundation

What it is, in plain words

Crawler access is the first binary gate in GEO: before retrieval, reranking, or citation can happen, an engine's bot must actually fetch your page and get a 200 with full HTML. The trap is that access has a chain of independent checkpoints (robots.txt, then your CDN/WAF, then any bot-management layer, then rendering), and any one of them can silently block a bot your robots.txt cheerfully allows. "Allowed in robots.txt" is a request, not a guarantee.

This node is the end-to-end verification discipline; the per-vendor fleet map (which OpenAI/Anthropic/Perplexity/Google bots do which job) lives in AI Crawlers. The stakes are high because the economics now pull toward blocking: Cloudflare measured crawl-to-referral ratios around 14:1 for Google but 1,700:1 for OpenAI, and ~71,000:1 for Anthropic's crawler, and since mid-2025 Cloudflare blocks AI crawlers by default on many new zones. The skill is making access an explicit, verified decision instead of an accident of your infrastructure's defaults.

The Access Chain

Four checkpoints, any of which can silently block you

the stated policy

robots.txt

Per-user-agent allow/deny rules: your intended policy. Name each bot explicitly rather than relying on wildcards, and split the decision: training bots are a licensing question, retrieval and user bots are a visibility question. See the per-bot map in AI Crawlers.

the silent blocker

CDN / WAF

Your CDN, firewall, or security plugin can 403 or challenge AI user-agents regardless of robots.txt. "Block bad bots" presets and default AI-crawler blocks (Cloudflare since mid-2025) are the most common cause of a brand vanishing from an engine with a clean robots.txt.

trust but verify

IP / bot verification

User-agent strings can be spoofed, so real bots publish IP ranges (or reverse-DNS) to confirm identity. Verification matters both ways: don't block a legitimate bot mistaken for a faker, and don't let a fake OAI-SearchBot skew your logs.

access ≠ arrival

The render handoff

A 200 with an empty JavaScript shell is a fetch that delivered nothing citable. Access only counts if the bot receives the actual content in the HTML, which is why this node hands directly to Rendering & Speed.

Myths vs Reality

Common misreadings, corrected

Myth"Our robots.txt allows the AI bots, so we're accessible."

Realityrobots.txt is only the first checkpoint. A WAF rule, a security-plugin preset, or your CDN's default AI-crawler block can 403 the very bots robots.txt allows, and it happens constantly. Access is what your server logs confirm was fetched, not what your file says is permitted.

Myth"Blocking AI crawlers protects our content at no cost."

RealityIt has a specific cost: retrieval and user bots are how you appear, cited, in the answers buyers read. Blanket blocks conflate training (a licensing decision, where the ~71,000:1 crawl ratios are a fair grievance) with retrieval (a visibility decision). Split them, or you delete yourself from ChatGPT and Perplexity to make a point about training.

Putting It to Work

Verify the whole chain, not just the file

The work is proving that access is real end to end (from stated policy through CDN, WAF, and rendering) and keeping it that way as your infrastructure changes underneath you.

The access-verification playbook

Confirm the policy in robots.txt

Fetch /robots.txt and read it: are the AI bots you care about named explicitly, with training vs retrieval decided deliberately? Fix accidental wildcards and legacy blanket blocks first; the per-bot reference is in AI Crawlers.

Fetch as each bot and check for a real 200

Use curl -A "OAI-SearchBot" (and each other user-agent) against your key pages. You're looking for status 200 and full HTML, not a 403, not a challenge page, not an empty shell. This single test catches most silent WAF/CDN blocks.

Audit the CDN / WAF / security layer

Check your CDN's AI-crawler settings (Cloudflare AI Crawl Control et al.), firewall rules, and any security plugin's bot presets. Since mid-2025 many defaults block AI bots out of the box; align these with your robots.txt so both tell the same story.

Verify bot identity by IP

Confirm real bots against their published IP ranges or reverse-DNS, so you neither block a legitimate crawler nor trust a spoofed one. This keeps your allow-list precise and your access logs honest.

Watch the logs: access is what they show

Pull 30 days of server/CDN logs and count hits per AI user-agent. Retrieval bots you allow should appear regularly; a bot going silent is your early-warning that a checkpoint changed. Policy is what the logs say, not what the file says.

The First Ranker Akash's build log · running example

The clean robots.txt that was blocking everything

A The First Ranker client insisted their access was fine: "robots.txt explicitly allows every AI bot, we checked." Their citations across ChatGPT and Perplexity had still flatlined. Akash skipped the file and went to the logs: zero hits from OAI-SearchBot, PerplexityBot, or ChatGPT-User in weeks. A curl -A test confirmed it: every AI user-agent got a Cloudflare challenge page (a 403 to a bot), because a "bot fight mode" toggle had been switched on during an unrelated security review.

The robots.txt had been telling the truth and lying at the same time: the policy said Allow, the edge said Deny. They carved out the retrieval bots in Cloudflare, re-ran the fetch tests until every allowed bot got a clean 200 with full HTML, and watched the logs. The crawlers returned within days; citations recovered over the next few weeks. The client's runbook now reads: "access is verified in the logs, never assumed from the file."

Verification Checks

How to know it's really done

0/4 verified

Your access is real when the logs and fetch tests agree with your policy. Click a check to mark it verified:

Update Cadence & Dependencies

Keeping it alive

Monthly logs · quarterly access audit

Monthly Scan logs for AI user-agent activity; investigate any allowed bot that went quiet.

Quarterly Re-run fetch tests and re-check CDN/WAF defaults; vendors change bot policies and edge providers change defaults without warning.

On infra changes Any new CDN, WAF, security plugin, or deploy pipeline can silently change access; re-verify the chain before assuming it held.

Works together with

Impact Weightage & Results TAT

What it moves, and how fast

GEO outcome7%

A binary gate: retrieval access is on or off, and one wrong checkpoint zeroes out every other GEO investment on the affected engine. Verifying the whole chain is the highest-ROI technical work there is.

SEO outcome3%

Mostly orthogonal to classic SEO, but the same log-and-fetch discipline catches Googlebot/Bingbot access problems that hurt both games.

Editorial estimate of this node's contribution to your total GEO / SEO outcome. Nodes overlap, so weights don't sum to 100.

Results TAT Days–2 weeks Confirming signalAllowed bots reappear in logs within days of a fix; citations on affected engines recover over the following weeks.

Tools, sources & live links continue below

Tools for this

Go deeper from Technical GEO

Voices to follow

Canonical reads

Always current

These links resolve live: what you get is generated or filtered the moment you click.

Ask Perplexity A cited answer, generated the moment you click Ask ChatGPT See how ChatGPT explains it today Google · past month Only what was published in the last 30 days Google News Latest coverage across publications YouTube · newest Most recent walkthroughs and talks