How do I check if my site is blocking AI crawlers?

Enter your URL above. We request that exact page as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot, and report what each one received: the page, a robots.txt block, an empty shell that needs JavaScript, or a refusal. There is no model involved — a crawler either got your page or it did not.

Which AI crawlers matter?

GPTBot trains and grounds ChatGPT; OAI-SearchBot fetches pages for ChatGPT Search; ClaudeBot serves Claude; PerplexityBot serves Perplexity; Googlebot feeds Google AI Overview and Gemini. Blocking any one of them removes you from that assistant’s answers entirely, and each is a separate line in robots.txt.

Why does my page work in a browser but fail this check?

Almost always JavaScript. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do not execute it — they read the HTML your server returns and nothing more. A React or Vue app that renders content on the client returns an empty shell to all four, so the page a person praises is a blank page to an assistant. Googlebot is the exception: it renders JavaScript on a second pass, which is why a site can appear in Google AI Overview and be invisible in ChatGPT.

How do I unblock AI crawlers in robots.txt?

Add an explicit Allow for each bot you want, by its exact product token — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot. A blanket User-agent: * rule applies to them only when no more specific block exists, and many sites carry an old Disallow aimed at scrapers that catches these too. Deploy, then re-run this check rather than assuming.

Does blocking AI crawlers protect my content?

It stops those crawlers fetching you, which also stops them citing you. That is a real trade and worth making deliberately: publishers who sell access often block on purpose. What is not deliberate is the common case — a rule copied years ago to stop scrapers, quietly removing a brand from every AI answer since.

Yes. No account, no card, and nothing stored. It is the same readability engine the paid report runs, on its own.

Free tool

Can AI crawlers read your site?

We request your page as each AI crawler and show you exactly what it received. No account, no card.

We fetch the page five times, once as each crawler, and read the HTML each one gets back.

What this checks — and why it is not a robots.txt reader

We make a real request for your page as each AI crawler and report what came back. Parsing robots.txt only tells you what a site says; a request tells you what a crawler gets. Those differ more often than anyone expects — a page can be perfectly allowed and still arrive empty.

No model is involved and there is nothing to interpret. An engine that cannot fetch your page cannot cite it, whatever else is true about your content.

The five crawlers, and which read JavaScript

Crawler	Feeds	Runs JavaScript
GPTBot	ChatGPT (OpenAI)	No
OAI-SearchBot	ChatGPT Search (OpenAI)	No
ClaudeBot	Claude (Anthropic)	No
PerplexityBot	Perplexity	No
Googlebot	Google AI Overview, Gemini	Yes, on a second pass

That last column is the one that catches people. Four of the five read only the HTML your server returns, so a client-rendered site is blank to every assistant except Google’s — which is exactly how a brand ends up visible in AI Overview and absent from ChatGPT.

How to check your site

1Enter the page you care about. Your homepage, or the page you would want an assistant to quote — a pricing or comparison page is often more revealing.
2We request it as five crawlers. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot, each with its real user agent, against your live robots.txt.
3Read the verdict per bot. Readable, blocked by robots.txt, needs JavaScript, or refused. They routinely disagree, and the disagreement is the finding.
4Fix the rule or the rendering. A robots.txt block is a one-line change. An empty shell needs server-side rendering or prerendering for the page that matters.
5Re-run it. Verify against the deployed site rather than the intention.

How to allow AI crawlers in robots.txt

Name each bot by its exact product token. A blanket User-agent: * rule applies to them only when nothing more specific matches — and plenty of sites carry an old Disallow written years ago to stop scrapers that catches these too.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

Should you block AI crawlers?

Blocking stops them fetching you, which also stops them citing you. That is a real trade and some publishers make it deliberately, because access is what they sell.

What is almost never deliberate is the common case: a rule copied in to stop scrapers, still in place, quietly keeping a brand out of every AI answer since. This check exists to tell those two situations apart.

Frequently asked questions

How do I check if my site is blocking AI crawlers?: Enter your URL above. We request that exact page as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot, and report what each one received: the page, a robots.txt block, an empty shell that needs JavaScript, or a refusal. There is no model involved — a crawler either got your page or it did not.
Which AI crawlers matter?: GPTBot trains and grounds ChatGPT; OAI-SearchBot fetches pages for ChatGPT Search; ClaudeBot serves Claude; PerplexityBot serves Perplexity; Googlebot feeds Google AI Overview and Gemini. Blocking any one of them removes you from that assistant’s answers entirely, and each is a separate line in robots.txt.
Why does my page work in a browser but fail this check?: Almost always JavaScript. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do not execute it — they read the HTML your server returns and nothing more. A React or Vue app that renders content on the client returns an empty shell to all four, so the page a person praises is a blank page to an assistant. Googlebot is the exception: it renders JavaScript on a second pass, which is why a site can appear in Google AI Overview and be invisible in ChatGPT.
How do I unblock AI crawlers in robots.txt?: Add an explicit Allow for each bot you want, by its exact product token — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot. A blanket User-agent: * rule applies to them only when no more specific block exists, and many sites carry an old Disallow aimed at scrapers that catches these too. Deploy, then re-run this check rather than assuming.
Does blocking AI crawlers protect my content?: It stops those crawlers fetching you, which also stops them citing you. That is a real trade and worth making deliberately: publishers who sell access often block on purpose. What is not deliberate is the common case — a rule copied years ago to stop scrapers, quietly removing a brand from every AI answer since.
Is this check free?: Yes. No account, no card, and nothing stored. It is the same readability engine the paid report runs, on its own.

Being readable is the floor, not the goal

Passing this check means an assistant can read you. Whether it names you when a buyer asks who the best option is — and which competitor it names instead — is a different question, and the one worth money.

Run a free AI visibility scan