What Is llms.txt and Do You Need It? An Honest Answer

Published April 30, 2026 · ClickRadius Institute

llms.txt is a proposed web standard: a markdown file at your site's root that gives AI systems a curated, token-efficient map of your most important content. It has been adopted enthusiastically by developer-documentation sites, dismissed publicly by Google representatives, and hyped well beyond the evidence by parts of the SEO industry. This article explains exactly what it is, what is actually known about who reads it, and a clear-eyed recommendation — because the honest answer to "do you need it?" is more interesting than yes or no.

Where llms.txt came from

The proposal was published in September 2024 by Jeremy Howard — co-founder of fast.ai and Answer.AI, one of the more respected figures in applied machine learning — at llmstxt.org. The premise is practical. Language models operate under context-window constraints: when an AI system wants to use your website at inference time, it cannot read all of it, and what it does fetch arrives as HTML cluttered with navigation, scripts, cookie banners and boilerplate that waste tokens and bury substance.

Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety. Converting complex HTML pages... into LLM-friendly plain text is difficult and imprecise.— llms.txt proposal, llmstxt.org

The proposed fix: a single markdown file at /llms.txt containing the site's name, a one-paragraph summary, and curated lists of links to the content that matters most — a hand-drawn map for machines, distinct from the exhaustive inventory of an XML sitemap. The proposal also describes an optional companion convention: markdown versions of pages (append .md to a URL) and an /llms-full.txt that inlines full content for documentation-style sites.

What the format looks like

The specification is deliberately minimal — plain markdown with a fixed skeleton:

# Acme Plumbing Co.

> Licensed plumbing contractor serving the Denver metro
> area since 2004. Emergency service, repiping, water
> heaters, and commercial maintenance contracts.

## Services

- [Emergency plumbing](https://acme.example/emergency): 24/7 response, licensed techs
- [Water heater replacement](https://acme.example/water-heaters): tank and tankless, same-week install

## Resources

- [Pricing guide](https://acme.example/pricing): typical cost ranges by job type
- [Service area](https://acme.example/areas): all 14 cities we cover

## Optional

- [Company history](https://acme.example/about): background and licensing details

The structural elements — one H1, a blockquote summary, H2-sectioned link lists with one-line descriptions, an "Optional" section for content that can be skipped under tight token budgets — are all the syntax there is. A competent implementation takes under an hour for most sites.

Who actually reads it: the evidence, honestly stated

This is where most coverage of llms.txt goes wrong in one direction or the other. The verifiable picture as of this writing:

Confirmed adoption on the publishing side

Thousands of sites have published llms.txt files, skewing heavily toward developer tools and documentation — Anthropic, for example, publishes both llms.txt and a full-content variant for its documentation, and documentation platforms such as Mintlify generate the files automatically for the sites they host. Community-maintained directories track hundreds to thousands of implementations. Publishing adoption is real and growing.

Unconfirmed adoption on the consuming side

No major AI engine — not OpenAI, not Anthropic, not Google, not Perplexity — has publicly committed to using llms.txt as an input to retrieval or ranking. Google's John Mueller said publicly in 2025 that no AI system he was aware of used it, comparing it to the old keywords meta tag. At the same time, server-log analyses shared across the industry do show AI-associated user agents fetching /llms.txt on some sites — fetching a file, of course, is not the same as weighting it.

Publish-side adoption is real; consume-side adoption is unproven. Any vendor telling you llms.txt will get you cited is ahead of the evidence — and any vendor telling you it is worthless is ignoring that the cost is one static file.— ClickRadius Institute analysis

What llms.txt is not

• It is not robots.txt. It grants no permissions and blocks nothing. Crawler access control lives in robots.txt, a standard with three decades of actual compliance behind it (RFC 9309).
• It is not a sitemap. An XML sitemap is an exhaustive machine inventory for indexing. llms.txt is the opposite: a curated shortlist. Dumping every URL into llms.txt destroys its one useful property, selectivity.
• It is not a substitute for structured data. Schema markup is consumed today by Google's documented pipelines; llms.txt consumption is speculative. If you have budget for one, the priority order is not close — see Schema Markup for AI Citation.
• It is not a ranking hack. There is no mechanism by which a text file makes weak content authoritative.

The case for doing it anyway

Given the unproven consumption, why does ClickRadius generate llms.txt files for the sites it manages — including this one? Four reasons, none of which require believing the hype:

1. Asymmetric cost. The downside is an hour of work and a few kilobytes. The upside, if any engine or agent framework formalizes consumption, accrues to sites that already have the file. Cheap options on plausible futures are good trades.
2. The agent trend points this way. The direction of travel in AI is toward autonomous agents that fetch and act on web content at inference time — a category that grew visibly through 2025 and 2026. Agents are exactly the consumers a curated, token-efficient site summary serves, and independent agent developers can and do read whatever helps them. Consumption does not require a Google announcement to begin at the margins.
3. The exercise itself has value. Writing a good llms.txt forces you to answer, in one page: what does this business do, and which ten URLs prove it? Most organizations cannot answer crisply. The artifact doubles as an editorial audit of your own information architecture.
4. Zero-click reality favors redundancy of machine-facing surfaces. With industry estimates putting zero-click searches around 60% and AI answer surfaces expanding — third-party trackers measured AI Overviews on roughly 15% of Google queries in early 2026 and climbing steeply since — betting on exactly one machine-readable channel is the risky posture, not the safe one.

How to write one that is actually good

1. Open with identity. H1 with the real business name; blockquote with a factual two-to-three sentence summary — what you do, where, for whom. No slogans. This paragraph may be the only thing a token-constrained system reads.
2. Curate 8–20 links, not 200. Core services or products, pricing, your most authoritative resources, contact. Each with a one-line description that would let a machine decide whether to fetch it.
3. Lead with pages that answer questions. If an AI is consulting your site, it is answering someone's question. Your best guides and FAQ-bearing pages belong above your press page.
4. Use the Optional section honestly for context that can be skipped under tight token budgets.
5. Keep it synchronized. A llms.txt pointing at dead URLs is worse than none — it demonstrates neglect to exactly the audience you meant to impress. Regenerate it when your site structure changes, ideally automatically.
6. Serve it as text/plain or text/markdown at the root, crawlable by everyone — do not block it in robots.txt, and do not gate it behind bot protection that challenges automated fetchers.

How llms.txt relates to the standards that already work

A useful way to place llms.txt is to line it up against the three machine-facing files the web already runs on:

• robots.txt (1994, RFC 9309 in 2022) answers "what may you fetch?" It took nearly three decades to go from convention to formal standard, but crawler compliance was widespread within a few years of its introduction because the major engines had operational reasons to honor it. Consumption drove adoption.
• XML sitemaps (2005) answer "what exists?" Sitemaps succeeded quickly for a different reason: Google, Yahoo and Microsoft jointly announced support at launch, so publishers knew from day one the file was being read. Announcement drove adoption.
• Structured data / schema.org (2011) answers "what does it mean?" Same pattern — a consortium of engines declared consumption first, and publisher adoption followed to roughly half the web over a decade.

llms.txt answers a genuinely new question — "what matters here, in a form a token-limited reader can use?" — but it is attempting the adoption path in reverse: publishers first, consumers maybe later. History says that is the hard direction, which is the honest structural argument for skepticism. The counter-argument is that the consumer side of this standard does not need to be a big-five announcement: the population of things that might read a curated site summary now includes every custom agent, RAG pipeline and automation built on LLM APIs — a long tail of consumers that never existed for robots.txt or sitemaps, any of which can adopt the convention without a press release.

Wherever that resolves, the ordering among the four files is not in dispute: the three proven standards answer questions engines are asking today, and llms.txt speculates about one they may ask tomorrow. Get robots.txt, the sitemap and structured data right first — then the speculative file costs you an hour you will not miss.

The verdict

Do you need llms.txt? No — nothing with unconfirmed consumption is a need. Should you have one? For most businesses, yes: the cost rounds to zero, the exercise sharpens your site's self-description, and it positions you for an agent-driven web that is arriving regardless of which standards win. Just sequence it honestly: after crawl access is correct, after structured data is complete, after your content carries the statistics, quotations and source citations that research — notably the Princeton-led GEO study (Aggarwal et al., KDD 2024) — shows measurably raise citation likelihood. llms.txt is the garnish on that plate, not the meal. For the full sequencing, see The Technical GEO Audit Checklist.

Frequently asked questions

Do ChatGPT, Gemini or Perplexity officially use llms.txt?

No major AI engine has publicly committed to consuming llms.txt as a ranking or retrieval input, and Google representatives have publicly downplayed it. Server logs across the industry do show AI-related crawlers fetching the file on some sites. Treat it as a low-cost hedge with unconfirmed upside, not a proven channel.

Does llms.txt replace robots.txt or my sitemap?

No. robots.txt controls crawler access, sitemaps enumerate URLs for indexing, and llms.txt is a curated guide to your most important content. They answer different questions and coexist; llms.txt grants no permissions and blocks nothing.

What belongs in a good llms.txt file?

An H1 with the site or company name, a short blockquote summary of what you do, then markdown sections linking your most authoritative pages with one-line descriptions each — core services, key resources, pricing, contact. Curate rather than dump: the file's entire value is selectivity.

ClickRadius generates and maintains llms.txt automatically — alongside the structured data, content signals and crawl configuration that carry more proven weight. See how machine-readable your site is today with a free AI Readiness Score, or view plans on the pricing page.