Technical plumbing

Do llms.txt and robots.txt actually matter for AI search?

By Arnav Mukherjee, founder of TofuBofu · July 6, 2026

Ship a website in 2026 and within a week someone will tell you to add an llms.txt file "so the AI can find you." It sounds official, it takes five minutes, and it feels like progress. I added one to TofuBofu early on for exactly that reason. Then I went looking for evidence that it did anything, and what I found was uncomfortable: for AI search, it mostly does not, at least not yet.

Meanwhile the file that genuinely decides whether AI engines can read you at all, robots.txt, is the one most people never check. Let me separate the hype from the plumbing that actually matters.

The short answer

If you remember one thing: make sure robots.txt is not blocking AI crawlers, and do not lose sleep over llms.txt. Now the detail, because the why matters.

robots.txt: the file that can silently erase you

robots.txt is an old, boring, load-bearing file. It sits at yoursite.com/robots.txt and tells crawlers which parts of your site they are allowed to fetch. The AI crawlers behave: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended all read it and obey it.

That obedience cuts both ways. If your robots.txt disallows those bots, whether you did it on purpose, a plugin did it for you, or an over-broad rule swept them up, you have quietly removed yourself from the set of pages those engines can read and cite. No error, no warning. You just are not in the answer, and you never find out why.

This is the highest-value thing on this page. Open your robots.txt right now and look for any Disallow that applies to GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or a blanket rule that blocks everything. If you want AI engines to recommend you, they have to be allowed in.

llms.txt: a good idea the engines have not adopted

llms.txt is a proposed file, written in clean markdown, meant to give a language model a curated summary of your most important pages, so it does not have to guess from your full site. The concept is sensible. The problem is that a standard only works if the engines actually read it, and right now they mostly do not.

The evidence is fairly blunt. As of 2026, the companies behind the major answer engines have not committed to reading llms.txt automatically. Google has publicly said it does not support it and is not planning to, with its own search staff comparing it to the long-discredited keywords meta tag. Ahrefs checked 137,210 domains, measuring the traffic they received in May 2026 and publishing the following month. 28 percent of them publish an llms.txt file, a figure Ahrefs itself tells you to treat as an upper bound because its analytics customers skew more technical than the web at large, and 97 percent of those files were fetched by nothing at all in the month measured. And a Semrush study found no statistical correlation between having one and performing better in AI results.

There is a real place llms.txt works today, and it is not AI search. Coding assistants like Cursor, Claude Code, and Copilot, MCP servers, and in-product AI helpers do fetch it, because they are pointed at a specific site and want a clean map of it. If that is your use case, ship one. If your goal is to be cited when a buyer asks ChatGPT for a recommendation, it is not the tool for the job yet.

Who is actually fetching the 3 percent

Ahrefs classified every user agent that hit an llms.txt file into twelve categories, and that table is the most useful part of the study, because it answers the question the headline does not. Of the small share of files that do get read, who is doing the reading?

Read the twelve categories individually and the largest is SEO audit tools at 21.7 percent of requests. Read them the way Ahrefs itself does, combining the four AI categories, and AI bots become the largest single bucket at 19.5 percent. Both statements are true. Quoting one without the other is how this file gets argued about, and the gap between the two framings is the actual finding.

Now the part that matters if you are about to quote the 21.7 percent. That category is generic SEO crawlers running routine site health checks, with no specific interest in llms.txt at all. And 2,334 of its 4,776 requests are Ahrefs' own crawlers, which is 10.6 percent of all traffic in the study. Strip those out and third-party SEO audit tools account for 11.1 percent. The largest single category is roughly half the study author's own bot, and Ahrefs says so in its own footnote rather than leaving you to find it.

The bucket that really is the industry inspecting the standard is 12.1 percent, and GEO/AEO tools are 5.8 percent of all requests on their own. That is a closed loop worth naming out loud: a measurable slice of all llms.txt traffic is commercial tools scanning to score whether you have an llms.txt, before anyone has established that a major answer engine reads one.

Then the result that settles the publish-one-just-in-case argument. Ahrefs found that zero AI bots went looking for llms.txt files that did not exist. Nothing is out there probing for the file and coming away disappointed, which means the cost of not having one is not a missed fetch. It is nothing.

One result cuts the other way, in favour of the file, and it lines up exactly with the paragraph above. Among AI bots, GPTBot is the top fetcher and Claude-Code is second, ahead of every AI search and assistant bot in the study. That is the coding-assistant use case showing up in the data. It is not evidence that llms.txt earns you a citation when a buyer asks an engine for a recommendation. It is evidence that the file does a real job for a real audience, and that the audience is developers and agents pointed at your documentation.

We generate an llms.txt for every brand we scan, built from the site's actual crawled sitemap rather than from a template. It is worth having, and it is worth roughly what this data says it is worth: hygiene, and a clean map for the tools that genuinely fetch it. It is not the thing that gets you named in an answer. Anyone selling it to you as the thing that gets you named in an answer is selling ahead of the evidence, and the evidence is public.

Where the hype comes from

llms.txt spread faster than its evidence because it is a tidy, hopeful idea and easy to write a confident blog post about. Adoption has grown, and you will see respected companies publishing one, which makes it look proven. But adoption by publishers is not the same as usage by the engines, and it is the usage that is missing. This may change. If the major engines start honoring it, the calculus flips overnight. Today, it is anticipation dressed as results.

What to actually do