Free tools / AI Crawler Checker
Free tool · robots.txt checker for AI bots
AI Crawler Checker
Enter a domain. We read your robots.txt the way each AI crawler reads it, then fetch your homepage as each crawler to catch the blocks robots.txt can't show: a CDN rule, a firewall, a bot challenge. Two columns, because the two are fixed in different places by different people.
Free, no signup, no email. Runs on your live site.
Sample result
What a result looks like
A real run on tofubofu.com, 10 November 2026.
Every AI agent we check is allowed by robots.txt, and none was refused at the door.
How it works
Two reads of the same door
We read robots.txt as each bot does
RFC 9309 is the standard: a crawler obeys the group that names it, and falls back to User-agent: * only when none does. Skip that fallback and a site blocking everything under * looks open.
We fetch your homepage as each crawler
Same address, the user-agent each vendor publishes. If the page loads for a browser and refuses GPTBot, something in front of your site is filtering by bot name.
We keep the two answers apart
A robots.txt line is edited in your CMS or code. A CDN or firewall rule lives in Cloudflare, your host or a security plugin. One column per place to look.
We say what each block costs
Each row carries the vendor's own description of what that agent feeds, linked.
Reference
The AI crawlers, and what each one feeds
| Agent | Owner | Job | Block it and |
|---|---|---|---|
| GPTBot | OpenAI | Training | OpenAI won't use your pages to train its models. ChatGPT search is a separate agent. |
| OAI-SearchBot | OpenAI | Search | Your pages can drop out of ChatGPT search answers and their links. |
| ChatGPT-User | OpenAI | User fetch | A user's request to read your page can fail. OpenAI says robots.txt may not apply. |
| ClaudeBot | Anthropic | Training | Anthropic won't train on your pages. Claude's search is a separate agent. |
| Claude-SearchBot | Anthropic | Search | Anthropic says your visibility in Claude's search results may drop. |
| PerplexityBot | Perplexity | Search | Your pages can drop out of the sources Perplexity cites. |
| Google-Extended | Control token | Google stops using your pages for Gemini training and grounding. Search, AI Overviews and AI Mode are unaffected. | |
| CCBot | Common Crawl | Training | You leave the Common Crawl archive, which many model makers train on. |
What to do with a result
What each result means, and what to do about it
| If the result says | It means | Do this |
|---|---|---|
| A search agent is blocked in robots.txt | That engine's search can't read your pages, so it can't cite them. | Delete the Disallow: / line under that agent, or add an Allow for the pages you want read. Then re-run the check. |
| A training agent is blocked | You've opted out of that vendor's model training. Search answers are unaffected. | Keep it if that's your policy. Make sure the matching search agent is still allowed. |
| Refused when fetched as a bot, loads for a browser | A CDN, firewall or plugin is filtering by bot name before robots.txt is read. | Open your CDN's bot or AI crawler settings and allow the search agents. Then run the check again. |
| Blocked through the wildcard group | You never named the bot, and your User-agent: * group shuts it out. | Add a group for each AI agent you want in, with an empty Disallow: line. |
| Everything allowed and loaded | Engines can reach you. That says nothing about whether they name you. | Run the free AI visibility scan to see whether they do. |
Why a CDN can block AI crawlers when robots.txt allows them
robots.txt is a request. A crawler that reads it then decides to comply. A CDN or firewall acts before any of that: it sees the request, matches the user-agent or the address, and returns a 403 or a challenge page. The crawler doesn't get as far as your robots.txt, so a robots.txt checker reports you open while the engine sees a wall.
On 1 July 2025 Cloudflare changed its default to block AI crawlers unless they pay creators for content. Hosts and security plugins ship their own rules too. That's why the fetch column exists.
This checks your site. It cannot tell you whether AI names you.
A free scan puts real buying queries to all five AI engines and shows who they name instead of you.
Check your AI visibility free →FAQ
Frequently asked questions
Which AI crawlers does this check?
Eleven agents from OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl, listed in the table above. We fetch the homepage as the seven that crawl on their own. The user-triggered fetchers and the two control tokens are read from robots.txt only.
What does it mean if a crawler is blocked?
That agent can't read your pages, and the cost depends on its job. A blocked training crawler keeps you out of future model training; a blocked search crawler keeps you out of that engine's live answers. Blocking training and allowing search is a legitimate policy. Blocking search by accident is the error this tool catches.
My robots.txt doesn't mention these bots. Is that bad?
No, unless your User-agent: * group disallows the whole site. A crawler with no group of its own follows the wildcard group, and if there is none, it's allowed. This tool shows which group each agent followed, so you can see the wildcard case instead of assuming it away.
Does robots.txt stop AI crawlers?
The major vendors document that their crawlers honour it. OpenAI and Perplexity say their user-triggered fetchers may not, because a person asked for the page. For the search crawlers that decide whether an engine can cite you, it works.
Why fetch the homepage as each bot?
Because robots.txt covers half the door. A CDN, firewall or security plugin can refuse a crawler before robots.txt is read, and a tool that only reads the file can't see that block.
A crawler was refused at the fetch. Is that proof my CDN blocks it?
Treat it as strong evidence and check your CDN settings. Some CDNs verify crawlers by IP address, and our request comes from our server, not OpenAI's or Anthropic's, so a verifying CDN can refuse us while it admits the real crawler. If the browser fetch was refused too, the site refuses automated requests in general.
Is Google-Extended the crawler for AI Overviews?
No. Google-Extended is a robots.txt control token with no crawler of its own. Google says it controls use for Gemini training and grounding and does not affect Search. AI Overviews and AI Mode read the Search index that Googlebot builds, so blocking Google-Extended doesn't remove you from them.
My crawlers are allowed. Why doesn't AI mention us?
Access is necessary and not sufficient. Being named also takes pages that answer buyers' queries and independent sources that back you up. The free scan shows where you're missing, engine by engine.
Sources
Sources and further reading
- RFC 9309, Robots Exclusion Protocol: how a crawler picks its group
- OpenAI: overview of OpenAI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User
- Anthropic: its crawlers: ClaudeBot, Claude-SearchBot, Claude-User
- Perplexity: crawlers: PerplexityBot, Perplexity-User
- Google: common crawlers: Google-Extended
- Cloudflare, 1 July 2025: the default changed to block AI crawlers