AI crawlers are bots that fetch web pages to feed AI systems, either as training data or as live sources for answers. The main roster: GPTBot collects training data for OpenAI, OAI-SearchBot retrieves pages for ChatGPT search, ClaudeBot crawls for Anthropic, PerplexityBot feeds Perplexity's live answers, and Google-Extended is a robots.txt control governing whether your content trains Gemini models.
The single most consequential fact about them: unlike Googlebot, they fetch raw HTML and do not execute JavaScript. A site can look perfect in Google and be a blank page to every AI crawler.
The roster and what each one does
- GPTBot (OpenAI): gathers pages for model training. Allowing it makes your content eligible to shape future ChatGPT base models.
- OAI-SearchBot (OpenAI): fetches pages when ChatGPT searches the live web. This bot controls whether you can be cited in ChatGPT answers.
- ClaudeBot (Anthropic): crawls content for Anthropic's Claude models.
- PerplexityBot (Perplexity): retrieves pages for Perplexity, which cites live sources on every answer. Blocking it removes you from its citation pool.
- Google-Extended (Google): not a separate crawler but a robots.txt token; disallowing it opts your content out of Gemini training while normal Googlebot search crawling continues.
They do not run JavaScript
GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot read the raw HTML your server returns and nothing more. Content assembled in the browser by React, Vue, or any client-side framework simply does not exist for them, while Googlebot, which renders JavaScript, sees everything. This is how sites end up healthy in Google and invisible to AI. The test takes one minute: view source on a key page and search for your product name, pricing, and main copy. If they are missing from the raw HTML, follow the fixes in making your site readable to AI crawlers.
The robots.txt trade-offs
Blocking AI crawlers is not one decision; it is several, and they cut differently. Blocking GPTBot or Google-Extended keeps your content out of training, which protects it from reuse but also erodes your brand's presence in future base models. Blocking OAI-SearchBot or PerplexityBot is more immediately costly for visibility: those bots power live citations, so blocking them removes your pages from answers your buyers see this week. Many publishers land on blocking training bots while allowing search bots. The major AI companies document that these crawlers honor robots.txt, so the file is your actual control surface; audit it before assuming anything about your AI visibility.
How to see who is visiting you
Your server or CDN access logs list every crawler by user agent, so search them for GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot to see who fetches what and how often. Pair that with the raw-HTML test above and you know both halves: who can visit, and what they can read. Whether all that crawling translates into your brand actually appearing in answers is a separate question, and checking what AI says about your brand covers it. Our free scan bundles an AI-crawler readability check with the visibility results.
Find out which bots can actually read your site
The free scan includes an AI-crawler readability check alongside your visibility results, so you can see in one pass whether the crawlers can read you and whether the answers mention you.
Frequently asked questions
Should I block AI crawlers on my site?
Separate the decisions. Blocking training crawlers like GPTBot or the Google-Extended token trades future base-model presence for content control. Blocking retrieval bots like OAI-SearchBot or PerplexityBot costs you live citations immediately. Brands that want AI visibility generally allow retrieval bots at minimum.
Do AI crawlers respect robots.txt?
The major named crawlers, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, are documented by their operators to honor robots.txt directives, and Google-Extended exists specifically as a robots.txt control. That makes robots.txt your primary lever for deciding which AI systems may use your content.
Does blocking GPTBot remove my brand from ChatGPT?
No. Blocking GPTBot only affects future training crawls of your own site; it does not retract anything already learned, and third-party pages about your brand remain in the corpus. ChatGPT search citations are governed separately by OAI-SearchBot, so you can block training and still be cited.
Which AI crawler matters most for visibility?
For near-term visibility, the retrieval bots: OAI-SearchBot and PerplexityBot, because they feed live, citable answers where change shows up in weeks. GPTBot and Google-Extended shape slower-moving training presence. If you allow only some bots, allow the retrieval ones.
Related answers