Technical plumbing
What is a headless browser, and why it decides whether AI can read your site
By Arnav Mukherjee, founder of TofuBofu · July 6, 2026
When I built the little crawler inside our scanner, I hit a wall that quietly explains a huge share of AI-invisibility. I would fetch a page, and for some sites I would get the full text, and for others I would get a nearly empty shell, even though the page looked perfect in my browser. The difference was not the site's quality. It was whether the content was already in the HTML, or painted in afterward by JavaScript.
That gap has a technical name, headless browsers, and it decides whether an AI engine can read you at all. It is worth understanding, because it is one of the few AEO problems that can make an otherwise-great site score zero.
A headless browser, in plain terms
A normal browser shows you a window. A headless browser is the same software running with no window at all. It loads a page, runs the page's JavaScript, and produces the fully rendered result, all automatically, in the background. It is how a machine can see a page the way a human sees it after everything finishes loading.
This matters because many modern sites do not put their content in the HTML directly. They ship a mostly empty page plus a bundle of JavaScript that fetches and draws the real content once it runs. To see that content, a crawler has to actually run the JavaScript, which means running a headless browser. And here is the split that decides your fate.
Google renders. AI crawlers do not.
The split-visibility problem
Googlebot has used a headless Chrome-based renderer for years. It runs your JavaScript and indexes the finished page, so a JavaScript-heavy site can still rank in classic Google search. The AI crawlers work differently. They fetch the raw HTML your server returns and move on, without running the page's code.
This is not a guess. An analysis of more than 500 million GPTBot fetches found zero evidence of JavaScript execution. When GPTBot hits a client-rendered app, it downloads the empty shell and leaves. The result is what analysts call split visibility: your site ranks fine on Google, whose headless browser rendered it, while being effectively blank to every AI engine, which never ran the code.
This single fact explains a category of frustrating cases: a beautiful, fast, modern site that is genuinely good and genuinely invisible in AI search. The content is there for humans. It was never there for the crawler.
The 30-second test
You do not need a developer to check this. On any important page, right-click and choose "view page source." That shows the raw HTML your server sent, before any JavaScript runs, which is what AI crawlers see. Now use find, and search that source for a distinctive sentence from your page.
If your text is in the source, AI crawlers can read it. If the source is mostly a short shell of script tags and your content only appears in the normal rendered view, your site is relying on client-side JavaScript, and the engines are likely missing your best content. That five-word test is the most useful thing in this article.
The fix
1. Run the view-source test on your top pages
Confirm your key text appears in the raw source, not only in the rendered view. Do this for your homepage, service pages, and any page you want cited.
2. Move to server-side rendering or static generation
Make sure the content is in the HTML when the page is fetched. On modern frameworks this is a setting; on a site builder it means confirming the platform outputs real HTML.
3. Keep critical facts in text, not scripts or images
Anything you want quoted, your services, locations, answers, should be plain text in the source, not injected later or locked inside an image.
4. Re-test after redesigns
A rebuild on a client-rendered framework can silently reintroduce the problem. The view-source test takes 30 seconds and catches it.
Worried your content is invisible to AI crawlers?
A free scan shows whether AI engines can actually read and cite your pages.
Get your free auditFrequently asked questions
What is a headless browser?
A headless browser is a real web browser that runs without a visible window. It can load a page, run its JavaScript, and produce the fully rendered result, but it does this automatically in the background instead of showing anything on screen. Search engines use headless browsers to see a page the way a human would after all the code runs.
Do AI crawlers use a headless browser to render JavaScript?
Mostly no. An analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution. The major AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, fetch the raw HTML the server returns and do not run the page's JavaScript. If your content only appears after JavaScript runs, those crawlers do not see it.
Does Google render JavaScript when AI crawlers do not?
Yes, and that gap is the whole problem. Googlebot uses a headless Chrome-based renderer that executes JavaScript and indexes the finished page. Most AI crawlers do not. This creates split visibility: a JavaScript-heavy site can rank on Google while being effectively blank to ChatGPT, Claude, and Perplexity.
Why does JavaScript rendering matter for AI visibility?
Because AI engines cite what they can read, and they read the raw HTML. If your key content is injected by JavaScript after load, AI crawlers receive a near-empty shell and have nothing to cite. Server-side rendering or static generation, where the content is already in the HTML on fetch, is what makes your pages visible to every crawler.
How do I check if my site relies on JavaScript rendering?
Open your page, right-click, and choose view page source, which shows the raw HTML before JavaScript runs. Search that source for a distinctive sentence from your page. If the text is there, AI crawlers can see it. If the source is mostly empty and your content only appears in the live rendered view, your site depends on client-side JavaScript and AI crawlers are likely missing it.
What is the fix for a JavaScript-rendered site?
Move to server-side rendering or static site generation so the HTML already contains your content when the page is fetched. On modern frameworks this is a configuration or hosting choice; on site builders it usually means confirming the platform outputs real HTML rather than a client-rendered app. The goal is that your important text exists in the source, not just in the rendered view.
Sources and further reading
- The rise of the AI crawler (Vercel): how AI crawlers fetch pages and their share of traffic.
- AI crawlers do not render JavaScript (Lantern): the 500M-fetch analysis showing zero JavaScript execution.
- Google JavaScript SEO basics: how Googlebot renders JavaScript, in contrast to AI crawlers.