Technical plumbing

AI crawlers explained: GPTBot, ClaudeBot, PerplexityBot, and who actually reads your site

By Arnav Mukherjee, founder of TofuBofu · July 13, 2026

When I first opened TofuBofu's own robots.txt to make sure AI engines could read us, I assumed there was one bot per company. One OpenAI crawler, one Anthropic crawler, done. Then I actually listed them out and found that OpenAI alone runs three different bots with three different jobs, and that allowing the famous one, GPTBot, says nothing about the other two.

Most advice about AI crawlers treats them as one blob: "allow the AI bots." That blur is where real mistakes hide, because the bots feed different surfaces on completely different clocks. Blocking the wrong one, or being blocked by a firewall you forgot you had, erases you from answers silently. Here is the actual map.

Three jobs, not one: training, indexing, and live fetching

Every AI crawler you will meet does one of three jobs, and the job determines how fast your content can show up in an answer.

Training crawlers collect pages to train future models. What they read today shapes what the model "knows" from memory after its next release, which is why this lane moves in months, not days. Search-index crawlers build the retrieval index an engine searches at answer time; pages here can start being cited in days to weeks. User-triggered fetchers grab your page live, mid-conversation, when a user's question makes the engine browse; they read whatever your site serves at that second.

Your site TRAINING CRAWLERS GPTBot · ClaudeBot Google-Extended (opt-out) feeds model memory · months SEARCH-INDEX CRAWLERS OAI-SearchBot · PerplexityBot Bingbot · Googlebot feeds retrieval · days to weeks USER-TRIGGERED FETCHERS ChatGPT-User · Claude-User Perplexity-User reads live, mid-answer · seconds The answer your buyer sees Each lane is controlled by its own robots.txt user-agent. Allowing one says nothing about the others.

The bots, engine by engine

OpenAI runs three. GPTBot collects pages for model training. OAI-SearchBot builds the index behind ChatGPT search, and it is the one that matters most if you want to be cited when ChatGPT searches the web. ChatGPT-User fetches pages on demand when a conversation triggers browsing. OpenAI documents all three and honors robots.txt rules for each separately. The practical consequence: a publisher who blocked GPTBot in 2023 to stay out of training data also stayed invisible in ChatGPT search until they noticed OAI-SearchBot was a different agent.

Anthropic runs ClaudeBot for training data collection, plus user-triggered fetching agents (Claude-User, and Claude-SearchBot for search). If Claude's answers about your category matter to you, ClaudeBot needs access today for it to matter at the next model release.

Perplexity runs PerplexityBot to build its index and Perplexity-User for live fetches. Perplexity is retrieval-first, it searches the web for nearly every answer, so index access converts to citations faster here than anywhere else. Worth knowing: Perplexity has faced credible reporting, from WIRED and later Cloudflare, that some of its fetching ignored robots.txt directives. Whatever you think of that, it cuts one way for a services firm: your public pages are likely being read regardless, so make them worth citing.

Google is the trickiest. Google-Extended is not a crawler you will see in logs; it is a robots.txt token that opts your content out of Gemini training and grounding. Google AI Overviews, meanwhile, is built from the normal search index that regular Googlebot crawls. So blocking Google-Extended does not touch your AI Overviews presence, and the only way out of AI Overviews is to restrict Googlebot itself, which also removes you from regular search. Almost nobody selling services should do that.

Microsoft's Bingbot feeds both Bing results and Copilot. It is the same crawler SEO has known for years, which means your existing Bing indexing already works toward Copilot answers.

What AI crawlers cannot do: run your JavaScript

Googlebot renders JavaScript. The AI crawlers, as a rule, do not. Vercel analyzed more than 500 million GPTBot fetches and found zero JavaScript execution. If your pages are client-rendered, a React or Vue app that assembles content in the browser, most AI crawlers receive the empty shell and move on. Your beautiful site reads as a blank page to the systems deciding whether to recommend you.

The fix is server-side rendering or static generation for every page you want cited. I wrote up the mechanics, and what a headless browser is and why the engines skip it, in the headless browser guide.

Are AI crawlers actually reaching your site?

A free TofuBofu scan checks your robots.txt and AI-bot access as part of the site audit, alongside where you show up in the answers themselves.

Run your free scan

The silent block: how firms vanish without knowing

The dangerous failure mode is not a deliberate block. It is the accidental one, and there are three common sources.

1. Over-broad robots.txt rules

A Disallow written years ago for a different problem, or a blanket rule from a plugin, quietly catches AI user-agents. Migrations and redesigns are the classic moment this happens: the new robots.txt ships with defaults nobody reviews.

2. CDN and firewall bot management

Security layers challenge or drop bots they do not recognize, and AI crawlers are new enough to be on the wrong list. Cloudflare now blocks AI crawlers by default for new sites, a setting many owners never revisit. Your robots.txt can say Allow while your firewall says no, and only the firewall wins.

3. Client-side rendering

Not a block in name, but the same result: the crawler is allowed in and finds nothing to read. If content needs JavaScript to exist, most AI crawlers never see it.

The check takes ten minutes: open yoursite.com/robots.txt and read it with the bot list above in hand, then filter your server or CDN logs for those user-agents to confirm the bots are not just allowed on paper but actually getting 200 responses. If GPTBot and friends appear in your logs regularly, you are being read. If they never appear at all, something upstream is turning them away.

Frequently asked questions

What is GPTBot?

GPTBot is OpenAI's training crawler. It collects publicly available web pages to help train OpenAI's models. It is not the bot behind ChatGPT search results: that job belongs to OAI-SearchBot, and on-demand page visits during a chat come from ChatGPT-User. The three are controlled separately in robots.txt, which is why blocking GPTBot alone does not remove you from ChatGPT search.

What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?

They do three different jobs. GPTBot gathers pages for model training, which shapes what the model knows months later. OAI-SearchBot builds the index that ChatGPT search retrieves from, which affects whether you can be cited in searched answers within days or weeks. ChatGPT-User fetches a page live when a user's conversation triggers browsing. Each obeys its own robots.txt rule.

Does blocking Google-Extended remove me from Google AI Overviews?

No, and this is one of the most common misunderstandings. Google-Extended is an opt-out control for Gemini training and grounding. AI Overviews is built from Google's normal search index, crawled by regular Googlebot. If you want out of AI Overviews you would have to restrict Googlebot itself, which also removes you from regular Google search. Blocking Google-Extended leaves AI Overviews untouched.

Should a B2B company block AI crawlers?

If you sell services and want to be recommended, almost certainly not. Blocking AI crawlers removes your site from the pool of pages AI engines can read and cite, and buyers increasingly start there: 51% of B2B buyers now begin research with an AI chatbot (G2, 2026). The block-by-default advice comes mostly from publishers whose content is their product. For a services firm, your pages are marketing, and you want them read.

Do AI crawlers execute JavaScript?

Mostly no. An analysis of more than 500 million GPTBot fetches by Vercel found zero JavaScript execution. Google is the exception because Googlebot renders JavaScript, which feeds AI Overviews. If your important content only appears after client-side JavaScript runs, most AI crawlers see an empty shell. Server-rendered HTML is the safe baseline.

How do sites block AI crawlers by accident?

Three usual ways: a robots.txt rule written for another purpose that sweeps up AI user-agents, a security or bot-management layer such as a CDN or WAF that challenges or drops unknown bots (Cloudflare now blocks AI crawlers by default for new sites), and site migrations or plugins that rewrite robots.txt without anyone re-checking it. The block is silent: no error, no notification, you simply stop appearing.

How do I check which AI crawlers are visiting my site?

Two places. First, your server or CDN logs: filter requests by user-agent strings like GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Bingbot. Second, your robots.txt and firewall settings: confirm none of those agents are disallowed or challenged. A free TofuBofu scan also includes an AI-bot access check as part of the site audit.

Sources and further reading

Related reading

Do llms.txt and robots.txt actually matter for AI search?
What is a headless browser, and why it decides whether AI can read your site
How does an AI engine index and parse your content?