Free tools / AI Crawler Checker

Free tool · robots.txt checker for AI bots

AI Crawler Checker

Enter a domain. We read your robots.txt the way each AI crawler reads it, then fetch your homepage as each crawler to catch the blocks robots.txt can't show: a CDN rule, a firewall, a bot challenge. Two columns, because the two are fixed in different places by different people.

Free, no signup, no email. Runs on your live site.

Sample result

What a result looks like

A real run on tofubofu.com, 10 November 2026.

Every AI agent we check is allowed by robots.txt, and none was refused at the door.

OAI-SearchBot: allowed, homepage loadedIts own group allows it. Your homepage loaded (HTTP 200) for OAI-SearchBot.
Claude-SearchBot: allowed, homepage loadedIts own group allows it. Your homepage loaded (HTTP 200) for Claude-SearchBot.
CCBot: allowed through the wildcard groupNo group names it, so it follows your User-agent: * group, which allows it.
Google-Extended: a control tokenNothing to fetch as. robots.txt is the only place it can be set.

How it works

Two reads of the same door

  1. We read robots.txt as each bot does

    RFC 9309 is the standard: a crawler obeys the group that names it, and falls back to User-agent: * only when none does. Skip that fallback and a site blocking everything under * looks open.

  2. We fetch your homepage as each crawler

    Same address, the user-agent each vendor publishes. If the page loads for a browser and refuses GPTBot, something in front of your site is filtering by bot name.

  3. We keep the two answers apart

    A robots.txt line is edited in your CMS or code. A CDN or firewall rule lives in Cloudflare, your host or a security plugin. One column per place to look.

  4. We say what each block costs

    Each row carries the vendor's own description of what that agent feeds, linked.

Reference

The AI crawlers, and what each one feeds

AI agents by owner and job, from each vendor's own documentation
AgentOwnerJobBlock it and
GPTBotOpenAITrainingOpenAI won't use your pages to train its models. ChatGPT search is a separate agent.
OAI-SearchBotOpenAISearchYour pages can drop out of ChatGPT search answers and their links.
ChatGPT-UserOpenAIUser fetchA user's request to read your page can fail. OpenAI says robots.txt may not apply.
ClaudeBotAnthropicTrainingAnthropic won't train on your pages. Claude's search is a separate agent.
Claude-SearchBotAnthropicSearchAnthropic says your visibility in Claude's search results may drop.
PerplexityBotPerplexitySearchYour pages can drop out of the sources Perplexity cites.
Google-ExtendedGoogleControl tokenGoogle stops using your pages for Gemini training and grounding. Search, AI Overviews and AI Mode are unaffected.
CCBotCommon CrawlTrainingYou leave the Common Crawl archive, which many model makers train on.

What to do with a result

What each result means, and what to do about it

Each result this tool can return, what it means, and the fix
If the result saysIt meansDo this
A search agent is blocked in robots.txtThat engine's search can't read your pages, so it can't cite them.Delete the Disallow: / line under that agent, or add an Allow for the pages you want read. Then re-run the check.
A training agent is blockedYou've opted out of that vendor's model training. Search answers are unaffected.Keep it if that's your policy. Make sure the matching search agent is still allowed.
Refused when fetched as a bot, loads for a browserA CDN, firewall or plugin is filtering by bot name before robots.txt is read.Open your CDN's bot or AI crawler settings and allow the search agents. Then run the check again.
Blocked through the wildcard groupYou never named the bot, and your User-agent: * group shuts it out.Add a group for each AI agent you want in, with an empty Disallow: line.
Everything allowed and loadedEngines can reach you. That says nothing about whether they name you.Run the free AI visibility scan to see whether they do.

Why a CDN can block AI crawlers when robots.txt allows them

robots.txt is a request. A crawler that reads it then decides to comply. A CDN or firewall acts before any of that: it sees the request, matches the user-agent or the address, and returns a 403 or a challenge page. The crawler doesn't get as far as your robots.txt, so a robots.txt checker reports you open while the engine sees a wall.

On 1 July 2025 Cloudflare changed its default to block AI crawlers unless they pay creators for content. Hosts and security plugins ship their own rules too. That's why the fetch column exists.

This checks your site. It cannot tell you whether AI names you.

A free scan puts real buying queries to all five AI engines and shows who they name instead of you.

Check your AI visibility free →

FAQ

Frequently asked questions

Which AI crawlers does this check?

Eleven agents from OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl, listed in the table above. We fetch the homepage as the seven that crawl on their own. The user-triggered fetchers and the two control tokens are read from robots.txt only.

What does it mean if a crawler is blocked?

That agent can't read your pages, and the cost depends on its job. A blocked training crawler keeps you out of future model training; a blocked search crawler keeps you out of that engine's live answers. Blocking training and allowing search is a legitimate policy. Blocking search by accident is the error this tool catches.

My robots.txt doesn't mention these bots. Is that bad?

No, unless your User-agent: * group disallows the whole site. A crawler with no group of its own follows the wildcard group, and if there is none, it's allowed. This tool shows which group each agent followed, so you can see the wildcard case instead of assuming it away.

Does robots.txt stop AI crawlers?

The major vendors document that their crawlers honour it. OpenAI and Perplexity say their user-triggered fetchers may not, because a person asked for the page. For the search crawlers that decide whether an engine can cite you, it works.

Why fetch the homepage as each bot?

Because robots.txt covers half the door. A CDN, firewall or security plugin can refuse a crawler before robots.txt is read, and a tool that only reads the file can't see that block.

A crawler was refused at the fetch. Is that proof my CDN blocks it?

Treat it as strong evidence and check your CDN settings. Some CDNs verify crawlers by IP address, and our request comes from our server, not OpenAI's or Anthropic's, so a verifying CDN can refuse us while it admits the real crawler. If the browser fetch was refused too, the site refuses automated requests in general.

Is Google-Extended the crawler for AI Overviews?

No. Google-Extended is a robots.txt control token with no crawler of its own. Google says it controls use for Gemini training and grounding and does not affect Search. AI Overviews and AI Mode read the Search index that Googlebot builds, so blocking Google-Extended doesn't remove you from them.

My crawlers are allowed. Why doesn't AI mention us?

Access is necessary and not sufficient. Being named also takes pages that answer buyers' queries and independent sources that back you up. The free scan shows where you're missing, engine by engine.

Sources

Sources and further reading