AI crawlers are automated bots operated by AI companies to fetch web pages, either to answer users' questions in real time or to collect data for training models.
Search bots and training bots are different
Most AI companies run separate bots for different jobs, and each respects its own robots.txt user agent:
| Company | Search / answer bot | Training bot |
|---|---|---|
| OpenAI | OAI-SearchBot (ChatGPT search); ChatGPT-User for user-requested fetches | GPTBot |
| Perplexity | PerplexityBot; Perplexity-User for user-requested fetches | None. Perplexity says it does not use PerplexityBot to train models. |
| Anthropic | Claude-SearchBot; Claude-User for user-requested fetches | ClaudeBot |
| Googlebot (AI Overviews and AI Mode use the Search index) | Google-Extended controls Gemini training use |
Sources: OpenAI, Perplexity, Anthropic, Google Search Central.


What to check
- Your robots.txt: allow the search bots if you want to be cited.
- Your CDN or firewall: bot-protection rules often block AI search bots by accident, even when robots.txt allows them.
- Your server logs: confirm the bots are actually reaching your pages.
Blocking a training bot does not stop you from appearing in that company's search answers, as long as its search bot is allowed.
See also llms.txt and retrieval-augmented generation.
