Capability gap · reference

Do AI chatbots crawl your website? Which bots, and how to check

Yes — several AI companies run named crawlers that fetch pages the same way Googlebot does, and most sites can check and control access to each one in robots.txt. If one of these is blocked, that assistant structurally cannot read your pages, independent of anything else about your content.

The main bots to know

User agent	Operator	What it's for
GPTBot	OpenAI	Fetches pages for ChatGPT's web-search and training pipelines
OAI-SearchBot	OpenAI	Specifically for ChatGPT's search-grounded answers
PerplexityBot	Perplexity	Fetches pages for Perplexity's answer engine
ClaudeBot	Anthropic	Fetches pages for Claude, including web-search features
Google-Extended	Google	Controls use of your content for Gemini and Bard-family models, separate from classic Googlebot search indexing
Bingbot	Microsoft	Classic Bing search indexing, also feeds Copilot's grounding in some configurations

This list changes as vendors add new agents — treat it as a snapshot, not an exhaustive or permanent registry.

How to check what your site currently allows

Open yoursite.com/robots.txt directly in a browser and look for Disallow rules under any of the user agents above, or a blanket User-agent: * block with no exceptions. A block can come from a deliberate choice, but it's often a default set by a CDN, security plugin, or page builder that the site owner never reviewed. For an automated check plus a generated, ready-to-publish config, use the AI crawler robots.txt generator and checker.

Robots.txt is a request, not a lock

robots.txt is an honor-system convention — a well-behaved crawler respects it, but it isn't a technical enforcement mechanism. Reputable AI crawlers from OpenAI, Anthropic, Perplexity, and Google publicly document compliance with robots.txt rules for their named agents; it's worth checking each vendor's current published policy if you need certainty for a specific one, since policies do change.

Should you block or allow them?

That's a judgment call outside anything we can answer generically — some site owners want their content excluded from AI training pipelines specifically while still allowing search/citation-oriented agents; others block everything; others allow everything. What we can say plainly: if your goal includes being cited by ChatGPT, Perplexity, or Gemini's grounded answers, blocking the corresponding search-oriented agent (OAI-SearchBot, PerplexityBot, Google-Extended) removes that possibility structurally, regardless of how good your content is.

Frequently asked

If I block GPTBot, does that stop ChatGPT from ever mentioning my site?

It stops ChatGPT's own crawler from fetching your pages directly. It doesn't guarantee your site never comes up in an answer if the underlying search index or a third-party source mentions you some other way, but it removes the primary direct-fetch path.

Does blocking these bots hurt my classic Google/Bing SEO?

Not directly — Google-Extended and GPTBot are separate from Googlebot and Bingbot, which handle classic search indexing. Check your robots.txt carefully so a blanket rule doesn't accidentally catch the wrong agent.

How do I know if these bots are actually visiting my site?

Server access logs will show the user-agent string on each request — filter for the names in the table above.

Related

Crawl access is one of several structural reasons a domain might not get cited — see the full diagnostic: Why doesn't ChatGPT recommend my business? For the companion machine-readable file: What is llms.txt and do you need one?