llms.txt Is Being Treated Like robots.txt, and That Is the Problem
Common Crawl found hundreds of thousands of llms.txt files, but many appear templated, empty, or full of crawler rules the format cannot enforce. Builders should treat the file as a context hint for AI systems, not an access-control layer.
TL;DR: llms.txt is useful as a machine-readable content guide, but Common Crawl’s findings show many sites are using it like an unenforceable robots.txt clone.
What did Common Crawl actually find?
The primary source here is Common Crawl’s analysis of 584,107 llms.txt files, covered by Matt G. Southern in Search Engine Journal’s “Some Sites Use llms.txt Like robots.txt, Common Crawl Finds.”
The headline is not that llms.txt is failing. It is that a lot of people seem unclear on what job it is supposed to do.
According to Southern’s write-up, Common Crawl found that many llms.txt files came from templates. Many contained no links. Some included crawler instructions similar to robots.txt, even though llms.txt does not enforce crawler behavior.
That matters because llms.txt has been pitched as a way to help language models understand a site. The useful version is simple: point AI systems toward canonical docs, policies, product pages, examples, and other high-signal content. Less guessing. Less scraping around the edges. Better context.
But if a site drops in a template file with no real links, that is theater. If it adds allow and disallow style rules, that is also theater unless crawlers voluntarily honor them, and unless the format they are reading even defines those semantics.
This is the same pattern we keep seeing with AI-era web standards. A loose convention appears. Platforms, CMS plugins, and SEO tools rush to support it. Site owners ship the file because it feels like responsible AI hygiene. Then the files become half metadata, half wish list.
Is llms.txt supposed to control AI crawlers?
Not really.
That is the key distinction. robots.txt is a crawler directive convention with decades of usage behind it. It still depends on crawler compliance, but at least the intent is clear: tell crawlers what paths they should or should not fetch.
llms.txt is better understood as context packaging. It tells an AI system, “If you want to understand this site, start here.” That is different from saying, “You may not crawl this page.”
The confusion is understandable. Site owners are dealing with AI bots, search bots, training data worries, content licensing questions, and hallucinated brand answers, all at once. A file named llms.txt sitting next to robots.txt practically invites people to treat it as another control panel.
But that mental model breaks fast.
If you want crawler restrictions, use the mechanisms crawlers actually check, including robots.txt, HTTP headers where supported, authentication, paywalls, and contractual controls. If you want better AI answers about your company, product, docs, or publication, use llms.txt as a curated map.
The difference is boring. It is also the whole point.
What should a good llms.txt file contain?
A useful llms.txt should probably be small, deliberate, and maintained.
Not a dump of every URL. Not a generic plugin artifact. Not a fake policy file.
For a product company, I would point it to the current docs, pricing page, changelog, API reference, terms, security page, and a few canonical explainers. For a media site, I would point it to editorial policies, author pages, topic hubs, evergreen explainers, corrections policy, and subscription information. For an open-source project, I would include install docs, contribution rules, release notes, governance, and examples.
The goal is not to maximize ingestion. The goal is to reduce ambiguity.
That is where Common Crawl’s finding about empty or templated files is most useful. It shows the difference between adoption and actual use. A file existing at the root path tells us almost nothing. A file with current, chosen links tells us the operator has made an editorial decision.
This is also where SEO muscle memory can hurt. The web has trained teams to ask, “What file do I need to add so the machine likes me?” AI context files need a different question: “What would I want an assistant to read before answering a customer about us?”
For builders, I would treat llms.txt as a lightweight context contract, not a shield. Add one only if you can maintain it. Keep it short. Link to canonical pages. Remove stale docs. Do not put crawler bans there and assume anyone will obey them. The catch most teams miss: the file is not for you, it is for the downstream answer. If the linked pages are vague, outdated, or marketing mush, llms.txt just helps models find the mush faster.