llms.txt: Meaning, Format, and What the Evidence Shows

llms.txt is a proposed Markdown file that a website publishes at its root, or under a section it controls, to give AI agents and LLM-based tools a short description of the site and curated links worth reading. It is a convention, not a ratified web standard. It does not control crawling, opt content out of training, or guarantee that any AI system will read it.

Publishing the file is easy. Knowing whether anything uses it is much harder, because the evidence collected so far measures requests for the file, not citations or rankings.

Where llms.txt Came From

Jeremy Howard proposed llms.txt in September 2024, announcing it through Answer.AI and publishing the specification at llmstxt.org. The goal was to give compatible tools succinct background on a site plus curated links to further resources, preferably in clean formats an agent can read without wading through page layout. Both pages come from the same author, so together they confirm the origin of the idea. They say nothing about how widely it has been adopted.

Howard and later implementers describe uses such as helping coding assistants and documentation tools find the right reference pages. Those are design goals. No source reviewed for this entry demonstrates token savings, faster responses, or more accurate answers caused by the file.

Proposal status in v2

The live proposal is labeled v2 and was modified on August 10, 2026, according to its changes page. It has not been ratified as a standard by a body such as the IETF or W3C. Some vendor documentation calls llms.txt an industry standard, but the specification itself presents a proposal, and that is the status this entry uses.

How an llms.txt File Is Structured

The format is plain Markdown that people and programs can both read. Version 2 keeps the required part very small.

The one required element

Only one element is mandatory: an H1 heading with the name of the project or site. A file with just that heading conforms to the proposal. It would tell a tool very little, though.

Optional summary, details, and file lists

After the H1, a publisher may add any of the following:

• A blockquote summarizing the site and the context a reader needs.
• Paragraphs or lists with more detail, written without headings.
• Sections introduced by H2 headings, each holding a list of files or pages.

Those H2 sections are optional. If you include a list of files, though, each item needs a Markdown hyperlink, optionally followed by a short note about the page. Loose structure overall, one strict rule inside lists. A fixed multi-section template is a publisher choice, not something the proposal requires.

What the Optional section means in v2

The proposal describes a section headed "## Optional" for secondary links. Earlier versions of the proposal tied that heading to tooling that could trim context mechanically. Version 2 retired those mechanics, as its changes page records, so the heading is now a convention for lower-priority material. No client is required to skip it.

Where the File Lives and How Tools Might Find It

Root files and subpath files

The proposal places the file at the root of a site, at /llms.txt. The v2 proposal also allows files under a subpath the publisher controls, such as a documentation folder, and suggests using the most specific file when more than one applies. That is a proposed rule. Whether a given agent follows it is a separate question that only testing can answer.

Proposed discovery signals

Version 2 also recommends offering clean Markdown versions of important pages and pointing to them, and to llms.txt itself, through alternate or describedby link relations in HTML or an HTTP Link header. These make resources easier to find for tools that look for them. They do not show that any particular crawler or assistant does.

llms-full.txt and other vendor variants

Mintlify, a documentation platform, can generate llms-full.txt, a combined file with fuller page content, along with other variants specific to its product. It is a reasonable pattern for documentation sites. It is also a vendor feature, and the core v2 proposal does not require it.

How the Intended Workflow Is Supposed to Work

The proposal expects agents to find the file or be given it, pick the relevant links, and fetch what they need. Laid out from publisher to reader, that becomes a chain of six steps, and only the first is under the publisher's control:

1. The publisher writes and hosts the file.
2. An agent or tool requests it, either on its own or because a person points it there.
3. The tool parses the Markdown.
4. The tool decides which listed pages to fetch.
5. Content from those pages shapes an answer or helps complete a task.
6. If the product shows sources, a page may be cited to the reader.

Howard's announcement notes that how the file is processed depends on the application using it. Nothing in the chain happens automatically because the file exists. A server log can show step two, but it cannot show steps three through six, which is why request counts are not proof of citations, rankings, or return on investment.

llms.txt Compared With robots.txt, sitemap.xml, and Structured Data

These files can look alike because they all sit on the technical side of a website. Their jobs are different.

RFC 9309 states that robots.txt rules are not a form of access authorization, and llms.txt offers even less control, so neither file is a security measure. Private content belongs behind authentication. llms.txt also does not replace robots.txt, the sitemap protocol, or structured data. The proposal positions it alongside existing standards, not in place of them.

What Google, Chrome, and AI Providers Have Documented

Google Search

Google's Search Central guide to optimizing for generative AI features, updated July 10, 2026, names LLMS.txt among special AI text files and says Google Search ignores them, so they neither help nor hurt visibility or rankings in Google Search. Google may still crawl the file like any other URL it discovers. That is ordinary crawling, not special support. The statement covers Google Search only, not other products or other companies.

Chrome Lighthouse audit

Chrome's Lighthouse tool includes an optional llms.txt audit for agentic browsing. A missing file that returns a 404 is treated as not applicable, while a server error is flagged. It is a technical check for browsing agents, not a Search ranking signal, and passing it does not show that any AI system benefits from the file.

Provider files and crawler documentation

Anthropic and Perplexity both host llms.txt files for their own developer documentation. That shows these companies use the format to describe their docs. It does not show that their products automatically read llms.txt files on other websites.

OpenAI's crawler documentation separates its crawlers by purpose, including OAI-SearchBot, GPTBot, and the user-triggered ChatGPT-User, without a general commitment to read llms.txt files. A request from one of those user agents is not evidence of how a file is used.

What the Log Studies Actually Measure

Three published studies offer usage data. All come from commercial companies, all rely on their own data, and none of the underlying logs have been independently reproduced. Read each number with its denominator.

Ahrefs: a large domain sample

Ahrefs checked the roots of 137,210 domains that received traffic and used Ahrefs Web Analytics in May 2026. About 28% of all sampled domains had a detected llms.txt file. Ahrefs itself cautions that its Web Analytics customers lean more technical and more SEO-aware than websites in general, so that share should not be read as adoption across all websites. The study also did not check whether the detected files followed the llms.txt specification correctly. Of the domains with a detected file, about 97% recorded no observed requests to the file that month. The 97% describes unrequested files within this sample. It is not a measure of adoption across the web.

Among requests to files that did receive traffic, Ahrefs classified about 96% as bots and 4% as humans by user agent, with named AI tools making up about 19.5% of those requests. These are shares of requests, not of domains, and they say nothing about citations. Metrics: share of sampled domains and share of requests. Geography: not specified by the source. Confidence: REPORTED.

EZY.ai: 83 sites over 12 weeks

EZY.ai, a company that sells llms.txt implementation, reported server logs from 83 sites with connected files between April 27 and July 19, 2026 (UTC). It counted 7 file fetches from the GPTBot family, 9 from ClaudeBot, 0 from PerplexityBot, 67 from Googlebot, and 193 from Meta-ExternalAgent. Crawler identity rests on user-agent attribution, which the study does not fully verify.

There is also an accounting gap. EZY describes a broader panel of 556 domains but itemizes 125 connected and 387 never-connected domains, which adds up to 512. The other 44 are not explained, and this entry does not try to fill that gap. Metric: file fetches by crawler user agent. Geography: not specified by the source. Confidence: REPORTED, with an unresolved panel discrepancy.

OtterlyAI: one site, 90 days

OtterlyAI reported 84 requests to /llms.txt among more than 62,100 requests it classified as AI-related on a single site over 90 days, or about 0.1%. One site, no control group, and no published raw data make this an observation, not a test of any effect. Metric: share of AI-classified requests on one site. Geography: not specified by the source. Confidence: REPORTED.

Within these samples and periods, request volumes were low or zero for many files and for several named AI crawlers. That is a narrow finding. None of these studies can tell you whether a file you publish will be parsed, followed, or reflected in an answer.

Should You Publish One?

There is no proven ranking or citation benefit, and Google says the file neither helps nor hurts Google Search. The decision comes down to cost, maintenance, and how the tools you care about actually behave.

When a file can make sense

A file is easiest to justify when you already maintain clean documentation that developers or agents use, such as API references or help centers, and someone can keep the index current. Treat any benefit as something to test, not something to assume.

Public content and maintenance

Everything in llms.txt is public, so never list pages, drafts, or internal resources you would not want found. The file protects nothing. Its links also go stale like any others, which means it needs an owner and a review schedule.

How to test behavior for your own site

• Confirm the file returns a normal success response. Lighthouse flags server errors.
• Log requests to the file and to the pages it lists, by user agent. A user-agent string alone does not prove who sent a request.
• Treat a fetch as retrieval, not as use in an answer.
• Track AI citations separately, and avoid crediting the file for changes that may have other causes.

How llms.txt Relates to GEO, LLMO, AEO, and AI Citations

llms.txt is a file. Generative engine optimization (GEO) is the broader work of earning visibility in AI-generated answers, and an optional file proposal is at most one small technical input within it. Large language model optimization (LLMO) and answer engine optimization (AEO) are also optimization approaches, not file formats, so none of them is a synonym for llms.txt.

An AI citation is an outcome you observe in an answer. Neither publishing a file nor having it fetched causes one automatically, which is why request logs and citation tracking should be kept apart. Each of these four terms has its own glossary entry, linked under Related Terms below.

Because Google Search ignores the file, visibility there still rests on standard search work, which Arfadia's SEO services page covers. For visibility in AI answers more broadly, see Arfadia's generative engine optimization services.

Frequently Asked Questions

What is llms.txt?

llms.txt is a proposed Markdown file that a website publishes at its root or a controlled subpath to give AI agents and LLM-based tools a short site summary and curated links to further resources. Jeremy Howard proposed it in 2024, and the current version is v2, modified in August 2026.

Is llms.txt an official web standard?

No. It is a proposal from one author, not a standard ratified by a body such as the IETF or W3C, even though some vendor documentation calls it one.

Does Google Search use llms.txt?

No. Google Search Central says Google Search ignores special AI text files such as llms.txt, so they neither help nor hurt rankings or visibility in Google Search. Google may still crawl the file like any other URL. This applies to Google Search only.

How is llms.txt different from robots.txt and sitemap.xml?

robots.txt states crawler preferences under RFC 9309, and sitemap.xml lists URLs for discovery. llms.txt is a curated Markdown overview with links. It does not control crawling, protect content, or opt a site out of AI training.

Does publishing llms.txt guarantee AI citations?

No. The available studies measure requests for the file, not whether a tool parses it, follows its links, uses the content, or cites the site. No source reviewed for this entry shows a proven effect on citations, rankings, or traffic.

What is llms-full.txt?

It is a combined file with fuller page content that some platforms, such as Mintlify, generate for documentation sites. It is a vendor implementation, not a required part of the core llms.txt proposal.

Can an llms.txt file live in a subfolder like /docs/?

Yes, under the v2 proposal, which allows files under a subpath the publisher controls and suggests using the most specific file that applies. Whether a given AI tool follows that rule has to be tested.

Sources

1. The /llms.txt file, Jeremy Howard, llmstxt.org, v2, published September 3, 2024, modified August 10, 2026
2. Announcement of the /llms.txt proposal, Jeremy Howard, Answer.AI, September 3, 2024
3. Changes, llms-txt project, v2 (August 2026)
4. Optimizing your website for generative AI features on Google Search, Google Search Central, updated July 10, 2026
5. llms.txt audit, Chrome for Developers, Lighthouse, updated May 5, 2026
6. RFC 9309: Robots Exclusion Protocol, RFC Editor, September 2022
7. Sitemaps XML format, Sitemaps.org
8. llms.txt, Mintlify documentation
9. Claude developer platform llms.txt, Anthropic
10. Perplexity developer documentation llms.txt, Perplexity
11. Overview of OpenAI Crawlers, OpenAI
12. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, Louise Linehan, with Xibeijia Guan, Ahrefs, June 15, 2026
13. We Put llms.txt on 83 Websites. OpenAI Read It 7 Times., EZY.ai, July 27, 2026
14. Llms.txt Experiment: What Marketers Get Wrong about llms.txt, Thomas Peham, OtterlyAI, last updated February 5, 2026

Related Terms

• Generative Engine Optimization (GEO)
• Large Language Model Optimization (LLMO)
• Answer Engine Optimization (AEO)
• AI Citation