Definition
AI Crawler
An AI crawler is a bot operated by an AI company to read web content, either to train models (e.g. GPTBot, Google-Extended) or to retrieve pages live when answering questions (e.g. OAI-SearchBot, PerplexityBot). Blocking these crawlers in robots.txt makes a site invisible to the corresponding AI systems — it can't be cited or recommended from content that was never read.
The main crawlers to know: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Gemini training), and meta-externalagent (Meta AI). Most AI crawlers do not execute JavaScript, so content that only renders client-side is effectively invisible to them.
Related terms
Reading definitions won’t tell you where you stand
Run the free check and see whether AI actually recommends your brand.
Check my AI visibility