Is my website blocked from AI crawlers?

This is the most common reason a business is invisible to AI assistants, and the one owners are least likely to know about, because nothing on the website looks wrong.

A blocked crawler produces no error you would ever notice. Your site loads normally in your browser. Your Google rankings are unaffected. Nothing in your analytics looks unusual. The only symptom is that AI assistants never mention you, which is easy to attribute to something else entirely.

The blocks are usually not deliberate. They arrive as a security default or a checkbox in a hosting dashboard, often labeled in language that sounds obviously good to agree to.

The two kinds of block, and why one hides from the other

There are two independent ways your site can refuse an AI crawler, and checking for one tells you nothing about the other.

A robots.txt block is a written instruction. Your robots.txt names a crawler and tells it not to crawl. Well-behaved crawlers obey it. You can read this file yourself at yourdomain.com/robots.txt.

A network-level block is a refusal. Your firewall or CDN sees the crawler's request and returns a 403 Forbidden before robots.txt is ever consulted. Nothing about this appears in robots.txt, so a file that looks perfectly permissive can sit behind a firewall that rejects every AI crawler that arrives.

This is why reading your robots.txt is not a sufficient check. You have to actually make a request as each crawler and record the status code that comes back.

Which crawlers to check

Not all AI crawlers matter equally. The distinction that counts is retrieval versus training.

Retrieval crawlers fetch pages at the moment someone asks a question, and they determine whether you can be cited. These are the ones to care about:

• OAI-SearchBot, which powers ChatGPT search
• ChatGPT-User, which fetches a page when ChatGPT opens a link mid-conversation
• ClaudeBot and Claude-User, for Claude
• PerplexityBot, for Perplexity
• Googlebot, which feeds Google's AI Overviews as well as ordinary search
• Bingbot, which feeds Copilot and serves as a fallback for other assistants

Training crawlers are a different decision

GPTBot, Google-Extended, CCBot, Applebot-Extended, Amazonbot, meta-externalagent and Bytespider are mostly gathering material for model training rather than answering a live question.

Blocking these is a legitimate choice, and it does not stop you appearing in AI answers. If you want your content excluded from training but still want to be recommended, block the training crawlers and permit the retrieval ones. That is a coherent position.

What is not coherent, and is usually accidental, is blocking retrieval crawlers while assuming you have only opted out of training. The two get bundled together by security products under a single friendly-sounding switch.

How to check it yourself

Read your robots.txt first, at yourdomain.com/robots.txt. Look for any of the crawler names above followed by Disallow: /. Also check whether the file exists at all, and whether it is actually a text file: some single-page sites answer every URL with the homepage, so robots.txt appears to exist while returning HTML.

Then test the network layer, which robots.txt cannot tell you about. Make a request to your homepage while identifying as each crawler and record the HTTP status. A 200 means the crawler got through. A 403 means your firewall refused it. If you have curl available, the pattern is a request to your homepage with the crawler's user-agent string set, checking the response code.

Do both, because they fail independently, and a site can pass one and fail the other.

If you find a block

For a robots.txt block, remove the Disallow lines for the retrieval crawlers, or better, add explicit Allow rules so your intention is unambiguous rather than merely unstated. If your host manages the file for you, turn that feature off first or your edit will be overwritten.

For a network-level block, the setting is in your CDN or security dashboard rather than in your site. On Cloudflare it is under bot management. Turn off the AI crawler blocking, then re-test to confirm the 403s have become 200s.

Re-test afterwards either way. Managed robots.txt features have a habit of reappearing after a platform update, which is a good argument for checking on a schedule rather than once.