What Is llms.txt and Should Your AI Company Add One?

Sometime in the last two years you have probably noticed a new file sitting quietly at the root of company websites, right next to the familiar robots.txt. It is called llms.txt, and the pitch behind it is simple and appealing: give language models a clean, curated summary of your site instead of making them chew through navigation bars and cookie banners. The question worth asking before you add one to your own domain is whether anything on the other end is actually reading it.

What llms.txt Actually Is

llms.txt is a plain Markdown file, proposed on September 3, 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai and a former president of Kaggle. The argument behind it is narrow and practical, not ideological: a language model cannot hold an entire website in its context window, and converting a normal HTML page into clean text is difficult and imprecise once you account for navigation, ads, and scripts. A single curated file at the site root, the proposal argues, can hand a model background information and a short list of links without any of that overhead.

The specification, maintained at llmstxt.org, is deliberately thin. Only one element is required, a top level heading with the site or project name. Everything else is optional: a one line blockquote summary directly under the title, a short paragraph of context, and any number of H2 sections that group links with brief descriptions. A section literally named Optional marks material an agent can skip when it is short on context budget. The proposal also describes a companion convention, llms-full.txt, which inlines the complete text of every linked page into a single file so an agent can load everything in one fetch.

The pitch behind llms.txt is essentially a card catalog for a website, a short, human curated index instead of the full shelf. Photo via Wikimedia Commons.

How It Differs From the Files You Already Know

llms.txt is often introduced as the robots.txt of the AI era, but the comparison only goes so far. Robots.txt is an actual internet standard, formalized by the IETF in 2022 after three decades of informal use, and it answers a permissions question, which bots may access which paths. Sitemap.xml answers a discovery question, listing every indexable URL for a search engine's crawler. llms.txt tries to answer a third, different question, not what is allowed and not what exists, but what matters most, curated by a human rather than inferred by a crawler.

A newer proposal adds a fourth category entirely. WebMCP, unveiled by Google at I/O 2026 and co-edited with Microsoft as a W3C community group draft, lets a page expose actual callable functions, like submitting a search or completing a checkout, directly to a browser based agent through a JavaScript API. Where llms.txt describes content, WebMCP describes actions. Early participants in Chrome's origin trial reportedly include Expedia, Shopify, and Etsy, and Google has been candid that it sees this as the more durable answer for agentic browsing, distinct from anything llms.txt was designed to solve.

Why People Are Adding It Anyway

The adoption case is not baseless. A number of real companies, including Anthropic's own documentation, Cloudflare, Perplexity, Stripe, Vercel, Zapier, Hugging Face, Supabase, and the documentation platform Mintlify, publish an llms.txt file today. An ecosystem of tooling has grown around it too: Yoast SEO can generate the file automatically for WordPress sites, Webflow supports uploading it directly to the site root, and Next.js, Astro, Docusaurus, and Mintlify all have plugins or built in support. And while no major AI platform has confirmed reading the file automatically, developer facing tools including Cursor, GitHub Copilot, and common retrieval frameworks do read it when a developer points them at a site. Mintlify, for its part, reported counting 436 visits from AI crawlers after adding the file to its own domain, a real but notably modest number for one of the format's most visible advocates.

Proponents also point to a practical, measurable benefit that has nothing to do with search rankings: serving clean Markdown instead of full HTML pages can cut the tokens an agent burns parsing a page dramatically, with some companies reporting reductions on the order of ten times fewer tokens per page, which translates into faster and cheaper agent runs regardless of whether any ranking system notices.

The file's real readers today are not search crawlers but developer pointed agents, coding assistants and retrieval tools directed at a specific site's docs. Photo via Wikimedia Commons, CC BY 2.0.

Who Is Actually Publishing One

Adoption has grown fast in raw numbers while staying small in relative terms. A year long tracking study by Originality.ai, monitoring more than 3 million websites, counted 4,088 llms.txt files in June 2025 and 36,120 by May 2026, an 8.8 times increase in twelve months. Including the companion formats, llms-full.txt and a rarer ai.txt variant, the total reached 38,980 sites, with the short llms.txt index accounting for 92.7 percent of all files found. The full content companion grew even faster from a much smaller base, from 23 sites to 2,463 over the same period.

The distribution across the web is stranger than the growth curve suggests. SE Ranking's 300,000 domain study found adoption remarkably flat across traffic tiers, about 9.9 percent among sites with under 100 monthly visits, 10.5 percent in the middle tier, and 8.3 percent among high traffic sites, meaning the biggest, best resourced sites are actually slightly less likely to publish the file than mid tier ones. Separate crawls tell the same story at the very top: a monthly scan of the Tranco top 1,000 domains found 8.7 percent publishing the file as of June 2026, and an HTTP Archive analysis put the top 10,000 figure at 5.61 percent, flat for two consecutive months.

Then, in early May 2026, one platform decision dwarfed all of that organic growth at once. Shopify quietly began serving llms.txt, llms-full.txt, and a newer agents.md file on every active storefront by default, with no announcement, no merchant email, and no opt in. The first official acknowledgment was a developer changelog entry on May 28, roughly three weeks after site owners started noticing the files on their own stores. The effect on the statistics was dramatic: one HTTP Archive analysis measured llms.txt on 78.1 percent of Shopify stores, and BuiltWith counts of live sites serving the file jumped to roughly 7.3 million, almost exactly matching its count of live Shopify stores. In other words, a large majority of all llms.txt files on the web today exist because one platform turned them on, not because millions of site owners chose to.

Shopify's silent May 2026 rollout put an llms.txt file on millions of storefronts overnight, instantly making e-commerce the single largest source of the file on the web. Photo via Wikimedia Commons, CC BY 4.0.

What Happens When You Actually Measure It

The measured picture is considerably less generous than the pitch. In a study published in June 2026, the SEO analytics firm Ahrefs examined 137,000 domains and found that only 28 percent had published an llms.txt file at all, and that 97 percent of the files that did exist received zero requests from any identifiable AI bot in the month studied. The breakdown of who did request the files is even more telling than the headline number. SEO audit tools led with 21.7 percent of all requests, followed by unidentified generic bots. Among actual AI systems, OpenAI's GPTBot accounted for 4.51 percent of requests, Anthropic's ClaudeBot for 0.80 percent, and DeepseekBot for 0.02 percent, while the retrieval bots that fetch pages to answer live user queries inside AI search products made up just 1.1 percent of requests combined. A separate study by SE Ranking, covering roughly 300,000 domains, found no statistical correlation between having an llms.txt file and how often a domain got cited in AI generated answers.

Google's own commentary has been unusually direct for a company that typically avoids ruling features out. Google's search advocate John Mueller has repeatedly compared llms.txt to the old keywords meta tag, a self written description that search engines learned to ignore decades ago because any site can claim to be the best result for anything. Mueller has said server logs show AI services largely do not even check for the file's existence, and that its one credible use so far is narrower than advertised, helping an agent that has already landed on a site navigate around it, not helping that agent decide which site to visit in the first place. Google engineer Gary Illyes confirmed at a Search Central event that Google does not support llms.txt and has no plans to. In a small, telling moment in December 2025, an llms.txt file briefly appeared on Google's own developer documentation site and was removed again the same day.

Server log studies, not marketing claims, are what settled the question of whether AI crawlers actually request the file. Photo via Wikimedia Commons, CC BY-SA 4.0.

THE VERDICT

Should Your AI Company Add One

For search visibility or AI citation frequency, the evidence points to no. Neither Google, OpenAI, Anthropic, nor Perplexity documents llms.txt as something their crawlers consume to decide what to cite, and the two largest independent studies available found essentially no traffic and no measurable correlation with outcomes. Treating it as an SEO or generative engine optimization tactic is, on the current evidence, a misunderstanding of what the data shows.

The practical recommendation follows from those two facts held together. Publish an llms.txt file if your audience includes developers and AI agents interacting with your documentation directly, keep it genuinely curated rather than dumping every URL into it, and revisit it whenever a linked page materially changes so it does not go stale. Do not publish one expecting it to move search rankings, win AI citations, or substitute for the fundamentals that the evidence still says matter, clear content, solid technical SEO, and being a source other people and other systems actually reference.

The strongest case for llms.txt is not search visibility, it is making your own documentation cheaper and faster for the AI tools your actual users already point at it. Photo via Wikimedia Commons, CC0.

Closing Assessment

llms.txt is not the robots.txt of the AI era, at least not yet. Its raw adoption numbers grew nearly ninefold in a year and then jumped by millions overnight when Shopify switched it on by default, but the files themselves remain largely unread, with two independent studies covering nearly half a million domains finding almost no AI bot traffic and no measurable citation effect. What survives that data is the narrow case: a small number of AI native and developer tool companies have found it genuinely useful for agent readability of their own docs. Add it for the narrow reason. Do not add it for the broad one.