Technical plumbing

Do llms.txt and robots.txt actually matter for AI search?

By Arnav Mukherjee, founder of TofuBofu · July 6, 2026

Ship a website in 2026 and within a week someone will tell you to add an llms.txt file "so the AI can find you." It sounds official, it takes five minutes, and it feels like progress. I added one to TofuBofu early on for exactly that reason. Then I went looking for evidence that it did anything, and what I found was uncomfortable: for AI search, it mostly does not, at least not yet.

Meanwhile the file that genuinely decides whether AI engines can read you at all, robots.txt, is the one most people never check. Let me separate the hype from the plumbing that actually matters.

The short answer

If you remember one thing: make sure robots.txt is not blocking AI crawlers, and do not lose sleep over llms.txt. Now the detail, because the why matters.

robots.txt: the file that can silently erase you

robots.txt is an old, boring, load-bearing file. It sits at yoursite.com/robots.txt and tells crawlers which parts of your site they are allowed to fetch. The AI crawlers behave: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended all read it and obey it.

That obedience cuts both ways. If your robots.txt disallows those bots, whether you did it on purpose, a plugin did it for you, or an over-broad rule swept them up, you have quietly removed yourself from the set of pages those engines can read and cite. No error, no warning. You just are not in the answer, and you never find out why.

This is the highest-value thing on this page. Open your robots.txt right now and look for any Disallow that applies to GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or a blanket rule that blocks everything. If you want AI engines to recommend you, they have to be allowed in.

llms.txt: a good idea the engines have not adopted

llms.txt is a proposed file, written in clean markdown, meant to give a language model a curated summary of your most important pages, so it does not have to guess from your full site. The concept is sensible. The problem is that a standard only works if the engines actually read it, and right now they mostly do not.

The evidence is fairly blunt. As of 2026, the companies behind the major answer engines have not committed to reading llms.txt automatically. Google has publicly said it does not support it and is not planning to, with its own search staff comparing it to the long-discredited keywords meta tag. One analysis of over 500 million AI-bot visits across 90 days found only a few hundred that fetched an llms.txt file at all. And a Semrush study found no statistical correlation between having one and performing better in AI results.

There is a real place llms.txt works today, and it is not AI search. Coding assistants like Cursor, Claude Code, and Copilot, MCP servers, and in-product AI helpers do fetch it, because they are pointed at a specific site and want a clean map of it. If that is your use case, ship one. If your goal is to be cited when a buyer asks ChatGPT for a recommendation, it is not the tool for the job yet.

Where the hype comes from

llms.txt spread faster than its evidence because it is a tidy, hopeful idea and easy to write a confident blog post about. Adoption has grown, and you will see respected companies publishing one, which makes it look proven. But adoption by publishers is not the same as usage by the engines, and it is the usage that is missing. This may change. If the major engines start honoring it, the calculus flips overnight. Today, it is anticipation dressed as results.

What to actually do