Which AI crawlers actually obey robots.txt
Read the four vendors' own documentation and the answer is not one yes or no per company. Three of them exempt user-initiated fetches from robots.txt and one does not, and blocking a company's training bot leaves its search bot running.
Every AI crawler guide answers this question with a single yes or no per company. Read the four vendors' own documentation and a sharper pattern appears: three of them exempt user-initiated fetches from robots.txt, and one does not. That distinction is the whole answer, and it is not stated anywhere on the first page of results.
The reasoning behind the exemption is consistent across the three. A fetch made because a person typed a question is treated as that person visiting the page, not as a crawler indexing it, and robots.txt governs crawlers. Whether you find that convincing is a separate matter from whether it is true, and it is true.
The agents, and what each vendor states
| Agent | What it does | Obeys robots.txt |
|---|---|---|
GPTBot | OpenAI, training data | Yes |
OAI-SearchBot | OpenAI, surfaces sites in ChatGPT search | Yes |
OAI-AdsBot | OpenAI, checks pages submitted as ads | Yes |
ChatGPT-User | OpenAI, visits a page when a user asks | No. OpenAI: rules "may not apply" |
ClaudeBot | Anthropic, training data | Yes |
Claude-SearchBot | Anthropic, search result quality | Yes |
Claude-User | Anthropic, visits a page when a user asks | Yes, unlike its counterparts |
PerplexityBot | Perplexity, search index | Yes |
Perplexity-User | Perplexity, visits a page when a user asks | No. "generally ignores robots.txt rules" |
Google-Extended | Google, opt out of Gemini training and grounding | Yes |
Google-Agent, Google-GeminiNotebook, GoogleMessages | Google, fetches a page for a user | No. Google: they "generally ignore robots.txt rules" |
Blocking one agent does not block the others
The second mistake in most robots.txt files is treating a company as a single crawler. Each vendor runs one agent for training, another for search, and a third for user-initiated fetches, and they are separately addressable. Disallowing ClaudeBot blocks training and leaves Claude-SearchBot and Claude-User untouched. Disallowing GPTBot does nothing to OAI-SearchBot.
That separation is useful rather than annoying, because it lets you make the distinction most site owners actually want: stay out of the training corpus, stay in the answers. Allow the search agents, decide about the training ones on your own terms, and understand that the user-initiated ones are not a decision you get to make on three of the four platforms.
This site's own file allows fourteen agents and disallows only /api/. Writing this post found a gap in it: OAI-AdsBot is not listed, because it did not exist when the file was written. That is the ordinary maintenance cost of a file like this, and the reason to re-read the vendor pages rather than a guide.
What robots.txt cannot do here
- It cannot stop a user-initiated fetch on OpenAI, Perplexity or Google. Only server-side controls can, and blocking those agents means a person who asks about your business gets an answer built without your page.
- It does not control whether you appear in Google's AI features. That is governed by the ordinary preview and index directives,
nosnippet,data-nosnippet,max-snippetandnoindex. There is no AI-specific opt-out. - It is not where
llms.txthelps. Google states that it ignores AI-text files in Search, including its generative features, and that they neither harm nor help. Keep one for other services if you like; do not count it as a lever.
Once the file is right, the question moves on to what those agents actually receive when they arrive, which is a different problem and a measurable one: see what an AI crawler actually receives from your site. The wider picture of how citations get made sits in how AI search engines decide what to cite and how ChatGPT picks which businesses to recommend.
