Which AI Crawlers Should You Allow? A robots.txt Guide

Training bots and answer bots do different jobs

OpenAI documents GPTBot as its training-focused crawler and OAI-SearchBot as the separate crawler used to surface websites in ChatGPT search; blocking GPTBot alone does not block OAI-SearchBot.

Anthropic documents three distinct crawlers: ClaudeBot for training, Claude-SearchBot for indexing content for search results, and Claude-User for fetching a page live when a person asks Claude a question about it. Each can be allowed or blocked independently.

Google documents Google-Extended as a token that controls whether content can be used to train Gemini and related models, entirely separate from Googlebot, which continues to govern standard Search crawling and AI Overview eligibility.

Perplexity documents PerplexityBot for crawling and indexing, and a separate live-fetch agent, Perplexity-User, that retrieves a page when a person asks Perplexity a question requiring it.

Sources: OpenAI: crawler documentation, Anthropic: crawler documentation, Google: crawler and fetcher overview, Perplexity: crawler documentation

Decide by objective, not by name recognition

Start from what is actually wanted, then pick the bots that match it. To opt out of training only while staying visible in answers, block the training-labeled bot (GPTBot, ClaudeBot, Google-Extended) and explicitly allow the retrieval or search bot for that vendor.

To opt out of a vendor entirely, block both the training bot and the retrieval or search bot, and accept that this removes any chance of being mentioned or cited by that system. To stay fully visible, allow both categories and rely on content quality and technical eligibility rather than blocking rules.

robots.txt is a signal you should verify, not a guarantee

Vendors document that their crawlers are built to respect robots.txt, but independent researchers have publicly disputed whether every crawler consistently does so in every case. Treat a permissive or restrictive rule as a stated policy to verify, not an enforcement mechanism.

Check server logs for the actual user-agent strings and IP ranges requesting pages, and compare that against what robots.txt allows. Also check CDN, WAF, and hosting-provider bot-management settings, since some of these block AI crawlers by default at the network layer regardless of what robots.txt says, which can silently remove a site from AI answers while the team believes access is open.

A starting point, not a template to copy blindly

A reasonable starting configuration disallows GPTBot, ClaudeBot, and Google-Extended, since these are the training-focused agents, while explicitly allowing OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User, since these are the retrieval and live-fetch agents that make a brand eligible to appear in that vendor's answers.

Review each vendor's current documentation before publishing the rule, since user-agent names and stated behavior can change, and adapt the split between training and retrieval access to the objective chosen above rather than copying a blanket rule.

Where this connects to the rest of your program

Crawler access is a prerequisite, not a strategy. A page that is well-written and fully indexable still cannot be cited if a firewall rule silently blocks the bot that would retrieve it.

This is exactly the kind of deterministic, verifiable check that should run after every deployment: research the gap, approve the access change, implement it, and confirm in production that the intended bots can actually reach the page. Rankout treats this as part of implementation validation rather than a one-time setup task.

What not to do

Do not block every AI crawler by default out of general concern, since that forecloses citation opportunities that may be wanted later. Do not assume blocking one vendor's bot affects a different vendor, and do not assume a documented “obeys robots.txt” policy is independently verified for a specific site without checking its own logs.