Free tool
Is your site blocking the AI crawlers?
Plenty of sites block AI crawlers without meaning to: a plugin default, a security setting, a line inherited from an old robots.txt. If a crawler cannot read you, the engine it feeds cannot cite you. This checks each one and tells you exactly what a block costs.
Free, no signup, no email. Runs on your live site.
This tool checks your plumbing. It cannot tell you if AI recommends you.
A free scan asks your buyers' real questions across all 6 AI engines and shows who gets named instead of you.
Run your free scanFrequently asked questions
Which crawlers does this check?
GPTBot and OAI-SearchBot (OpenAI training and ChatGPT Search), ChatGPT-User (live fetches when a user follows a link), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini and AI Overviews training use, separate from Googlebot) and CCBot (Common Crawl, which many models train on).
What does it mean if a crawler is blocked?
That surface cannot read your pages. The consequence differs by bot: blocking GPTBot keeps you out of future model memory, blocking OAI-SearchBot removes you from ChatGPT's live search answers, blocking PerplexityBot removes you from Perplexity citations. Blocking is a legitimate choice for some businesses; the point is to make it a decision rather than an accident.
My robots.txt does not mention these bots at all. Is that bad?
No. Absence of a rule is permission, not denial: a crawler with no matching directive is allowed. You only need explicit lines if you want to allow something your wildcard rule blocks, or to block a specific bot.
Does robots.txt actually stop them?
The major AI crawlers documented here publicly honor robots.txt. It is a convention rather than an enforcement mechanism, so a badly-behaved scraper can ignore it, but for the engines that matter for AI visibility it works.
My robots.txt is fine but AI still does not mention us. Why?
Access is necessary, not sufficient. Being crawlable only means an engine can read you; being cited also needs content structured for extraction and corroboration from sources the engine trusts. A free scan shows which of your buyers' questions you are actually missing from.
Can a CDN or firewall block crawlers even if robots.txt allows them?
Yes, and it is a common cause. Cloudflare and similar bot-protection settings can challenge or drop AI crawlers before robots.txt is ever consulted. If this tool shows everything allowed but you still see no AI traffic, check your CDN's bot rules next.
Other free tools