Why AI bots can't crawl your site (and how to fix it)

What “blocked” actually means for AI answers

Answer engines — ChatGPT, Perplexity, Google AI Overviews, Claude — don't invent facts about your business. They fetch your pages, read them, and cite them. If their crawler can't reach your pages, you simply don't exist in the answer. No error is shown to you; the citation just goes to a competitor whose pages loaded.

This is the core of AEO (Answer Engine Optimization): being crawlable by the bots that power AI answers. And the most common reason a site fails it isn't a missing llms.txt or thin structured data — it's that the crawler was turned away at the door, before it ever saw the HTML.

robots.txt is a request; the server is the wall

There are two completely different places an AI bot can be blocked, and they behave nothing alike.

robots.txt is a polite request. It's a text file that says “please don't crawl these paths.” Well-behaved crawlers — GPTBot, PerplexityBot, ClaudeBot, Google-Extended — read it and obey. But it's voluntary: it blocks nothing technically, and it lives or dies on the crawler's goodwill.

A server- or WAF-level block is a wall. Your host, CDN or firewall inspects the incoming request's user-agent and returns a 403 Forbidden, a 429, or a challenge page before your site ever runs. The crawler gets nothing. Crucially, this can happen while robots.txt cheerfully says “allowed” — the two systems don't know about each other.

That gap is where most silent AI-invisibility lives: you (or a plugin, or a tutorial) allowed the bots in robots.txt, but the server blocks them anyway — and nothing warns you.

Why hosts and WAFs block AI bots in the first place

Blocking AI crawlers is rarely a decision you consciously made. It usually arrives as a default:

Bandwidth and CPU. AI crawlers can be aggressive — a training crawler may request thousands of URLs a day. Budget and shared-hosting providers increasingly ship user-agent blocklists that drop AI bots to protect their infrastructure, and they don't always tell you.

One-click CDN toggles. Cloudflare's managed “Block AI bots” setting is a single switch. It's easy to enable thinking you're only stopping training scrapers — but the same rule blocks GPTBot, ClaudeBot, Amazonbot, CCBot, Bytespider and more, including the crawlers that would otherwise cite you.

Security plugins and firewall rules. WordPress security plugins and WAF rulesets often bundle “bad bot” lists that sweep up AI crawlers alongside genuine scrapers. A rule written to stop content theft quietly also stops citation.

The result is the same: a block you never intended, saving a little bandwidth at the cost of every future AI citation.

How to tell if your site is blocking AI bots

You can test it yourself with one request per crawler — ask for your homepage while presenting the bot's user-agent, and watch the status code.

From a terminal: curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" https://yoursite.com/ — a 200 OK means the bot gets in; a 403, 429, 451 or 503, or a Cloudflare “Just a moment…” challenge, means it's blocked.

Run it for each crawler you care about (GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot). If a normal browser user-agent gets 200 but the bot user-agents get 403, the block is user-agent-based — the common, cheap kind, and the easiest to fix. If even Googlebot is blocked, your host is turning away crawlers wholesale, which hurts classic SEO too.

One caveat: this tests user-agent-string blocking. Sophisticated blocks that verify a crawler by its published IP ranges can't be reproduced from your laptop — but those are rarer, and a user-agent block is what you'll find the vast majority of the time.

How to unblock the crawlers you want

Fixing this is about the wall, not robots.txt — editing robots.txt does nothing if the server returns a 403. Go to whatever layer is doing the blocking:

Cloudflare / CDN. Turn off the managed “Block AI bots” rule, or replace it with a custom rule that allows the citation crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) while still challenging the rest. Cloudflare also lets you rate-limit instead of block, which caps the bandwidth without going dark.

Host firewall / ModSecurity. Ask your host (or check the firewall rules) for a user-agent blocklist that includes GPTBot, ClaudeBot or PerplexityBot, and allowlist the ones you want to be cited by.

Security plugin. In your WordPress security plugin's bad-bot or user-agent blocking section, remove the AI citation crawlers from the blocklist.

Then decide who you actually want. Blocking is a legitimate choice for training crawlers — allowing them contributes your content to model training. But citation crawlers are different: blocking GPTBot, OAI-SearchBot, PerplexityBot or ClaudeBot removes you from live AI answers with no upside. A common, defensible stance is to allow the citation and search crawlers while blocking or rate-limiting the pure training ones — OpenAI and Anthropic both support this split, so you can allow OAI-SearchBot for search while disallowing GPTBot for training.

Whatever you choose, re-test with the curl command above afterwards to confirm the server actually serves the bots — don't assume the toggle worked.

How Relvato monitors AI-crawler access for you

Testing once by hand is fine; the problem is that this block comes back. A CDN plan change, a security-plugin update, a new host firewall default, or a colleague flipping the Cloudflare switch can re-block AI crawlers months later — silently, exactly like the first time.

Relvato's AEO readiness check runs this test on every check. It requests your homepage as each answer-engine crawler (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) and confirms the server actually serves them — not just that robots.txt allows them. A browser-user-agent control avoids false alarms, and a Googlebot control tells AI-specific blocking apart from wholesale bot-blocking.

Because it runs continuously and re-checks after site updates, you find out the moment a deploy or a setting change starts turning AI crawlers away — with an alert, before weeks of lost citations go by. Alongside it, the same check verifies robots.txt policy, llms.txt, structured data, a sitemap, snippet metadata and server-rendered content — so “can AI find and cite me?” is answered end to end. It runs on any site with a verified domain, no plugin required.