SEOCompany.com Start your analysis
ServicesServices overviewNational & International SEOAI Search VisibilityEcommerce SEONews SEO for PublishersLocal SEO & Google Business ProfileReputation Management
InsightsInsights overviewAll articlesSEO GlossaryImportant SEO TermsLocal SEO articlesEcommerce articlesSEO tools
CompanyCompany overviewOur HistoryLocationsChoosing an SEO companyEditorial principlesOwnership & fundingContact
Pricing Start your analysis 800-724-0252
Small inspection drones fly low over a track at night, teal lights glowing
  1. Home
  2. SEO Glossary
  3. AI Crawlers
SEO Glossary · A

AI CrawlersNEW

DefinitionReviewed Nov 19, 2026All terms →

AI crawlers are automated bots operated by AI companies to fetch web pages, either to answer users' questions in real time or to collect data for training models.

Search bots and training bots are different

Most AI companies run separate bots for different jobs, and each respects its own robots.txt user agent:

CompanySearch / answer botTraining bot
OpenAIOAI-SearchBot (ChatGPT search); ChatGPT-User for user-requested fetchesGPTBot
PerplexityPerplexityBot; Perplexity-User for user-requested fetchesNone. Perplexity says it does not use PerplexityBot to train models.
AnthropicClaude-SearchBot; Claude-User for user-requested fetchesClaudeBot
GoogleGooglebot (AI Overviews and AI Mode use the Search index)Google-Extended controls Gemini training use

Sources: OpenAI, Perplexity, Anthropic, Google Search Central.

An inventory robot scans warehouse shelves at night
A technician checks fiber cables between server racks

What to check

  • Your robots.txt: allow the search bots if you want to be cited.
  • Your CDN or firewall: bot-protection rules often block AI search bots by accident, even when robots.txt allows them.
  • Your server logs: confirm the bots are actually reaching your pages.

Blocking a training bot does not stop you from appearing in that company's search answers, as long as its search bot is allowed.

See also llms.txt and retrieval-augmented generation.

The long run starts with one call

SEO is a marathon, not a sprint.

Tell us the searches you need to win. We'll tell you honestly whether they're winnable, what it takes and how long it will take. We don't guarantee rankings or AI citations. Firms that do are worth avoiding.

Start your analysis →800-724-0252Mon–Fri · 9–5 ET