What an llms.txt file actually is

llms.txt is a plain text file written in Markdown, served from the root of your domain at /llms.txt. Its job is to hand an AI agent a short, curated map of your site so the agent does not have to crawl and parse your navigation to find the pages that matter. The specification calls this “concise, expert-level information gathered in a single, accessible location”.

The format, set out in the llms.txt specification, is deliberately small. You need an H1 with the name of the site or project. Below it, a blockquote holding a one-paragraph summary. Below that, H2 headings that group lists of Markdown links, each link optionally followed by a colon and a short note about what the page contains. That is the whole specification. A section headed Optional is a convention for links an agent can skip when it needs a shorter context, and version 2 removed the mechanical meaning that heading carried in version 1.

Version 2, published on 10 August 2026, added the part the original spec lacked: a documented way for an agent to find the Markdown version of an individual page. The spec recommends serving a clean Markdown copy at the same URL with .md appended or substituted, and pointing to it with rel="alternate" type="text/markdown", alongside rel="describedby" for the llms.txt file that covers the page. Both can be sent as HTTP Link headers rather than markup.

Which AI engines have confirmed they read it

This is the question the post you are reading exists to answer, and the honest answer in September 2026 is short. None of the major AI search engines documents fetching llms.txt from your website.

Google is the most direct. Its Search Central documentation on AI features states that “there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary”, and then removes any ambiguity: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” John Mueller has publicly compared the file to the keywords meta tag, and Google representatives have said at Search Central events that Google does not support it and has no plans to.

OpenAI’s publisher documentation covers four user agents, GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, and tells webmasters to manage access through robots.txt directives and published IP ranges. It links to an llms.txt file, but that file is OpenAI’s own documentation index for agents reading OpenAI’s docs. Publishing one is not the same as reading yours. Anthropic’s crawler documentation lists ClaudeBot, Claude-User and Claude-SearchBot, and covers robots.txt and IP verification. It does not mention llms.txt at all. Perplexity documents PerplexityBot and Perplexity-User, and again the control surface is robots.txt.

Twelve documented user agents on one side. Zero documented consumers of llms.txt on the other. That asymmetry is the practical case for spending your hour on robots.txt and crawler settings for AI before you spend it on a Markdown index.

Google says two different things, depending on which product you ask

The confusion in the market is real, and it has a cause. Google Search says you do not need the file. Google Chrome checks for it. The Lighthouse team added an llms.txt audit inside a new Agentic browsing category, describing llms.txt as “an emerging convention used to provide a machine-readable summary of a website’s content, specifically designed for LLMs and AI agents”. The audit shipped in Lighthouse 13.3.0 and reached stable Chrome DevTools with Chrome 150.

Read what the audit does before you read anything into it. Lighthouse flags a page only if a server error occurs while fetching llms.txt. If the file is absent and the server returns a clean 404, the audit is marked Not Applicable, “as providing the file is optional at the moment”. That is a check for a broken response, not a recommendation to publish. The agentic browsing category itself is experimental, and Google has described it as informational and unbenchmarked.

So a Lighthouse report that mentions llms.txt is not Google Search asking you for one. Two teams inside the same company are looking at two different futures: one where you are found through the index, and one where an agent arrives at your site already knowing where it wants to go.

Where llms.txt genuinely gets used today

Dismissing the file entirely is as wrong as overselling it. There is a real audience, it is just not the one the hype posts describe.

The reader is an agent that already has your URL. A coding assistant told to use your API, a research agent handed your domain, an internal tool pointed at your documentation. For those, a curated Markdown index is genuinely faster and cheaper than parsing a JavaScript-rendered docs site. This is why the AI labs publish llms.txt files for their own developer documentation, and why documentation platforms generate one automatically. The spec’s own version 2 examples were rewritten around this use, coding agents fetching docs, rather than the speculative scenarios of the original.

What llms.txt does not do is get you discovered. Discovery still runs through crawling and indexing, which is why structured data, internal linking and entity clarity keep earning their place in a generative engine optimisation plan while llms.txt waits for an adopter.

llms.txt and robots.txt are not alternatives

These two files get bundled together constantly, and they do opposite jobs. robots.txt is a permission layer. It tells a named crawler what it may fetch, and the AI companies have built their opt-out mechanisms on top of it. Google-Extended, for example, governs whether your content is used for training and grounding in Gemini Apps and the Vertex AI API, and Google states plainly that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”.

llms.txt is an invitation layer. It assumes access has already been granted and suggests what to read first. It carries no enforcement, no directive syntax, and no standing with any crawler that has not agreed to honour it. If your concern is AI companies taking content without permission, llms.txt does nothing for you, and the crawler settings post linked above is where to start. If your concern is compliance, data privacy in AI search is a separate matter again.

The way to find out which bots are actually on your site is to look. Server log analysis for AI bot traffic will show you GPTBot, ClaudeBot and PerplexityBot arriving, and it will also show you how often anything requests /llms.txt. On most UK business sites, that second number is close to zero.

What to put in yours, if you publish one

The cost of a good llms.txt file is about twenty minutes, and it does no harm. If you want one, build it properly rather than dumping a sitemap into Markdown.

Start with an H1 carrying the trading name you want engines to use, matching the name on your site, your Companies House record and your directory listings. Add a blockquote of one or two sentences that states what the business does, where it operates and who it serves. Then group links under H2 headings that reflect how a buyer thinks: services, pricing, evidence, contact. Annotate each link with a short note, because the note is the part an agent reads when deciding whether to fetch the page.

Keep it to the pages you would want quoted. Twenty good links beat two hundred. Update it when the underlying pages change, because a stale index is worse than none, and update cadence already matters for the surfaces that do read your pages. If you want agents to reach clean Markdown copies, follow version 2 and expose them at page.md with the alternate link relation. Then stop, and put the rest of the hour into the content itself, which is what ranking in ChatGPT still turns on.

What would count as evidence that it works

Plenty of pages published in the last year claim llms.txt lifts AI citations. None of them shows a controlled result. The reason is that the file’s effect, if it has one, is invisible to the measurement tools most sites own, and the honest position is to say so rather than to imply a causal link from a rising chart.

Real evidence would look like three things. First, an engine’s own documentation stating that it fetches the file and what it does with it. Second, server logs showing repeated requests to /llms.txt from a named AI user agent, not from a browser or a scraper. Third, a split test across comparable pages where the only variable is the file, run long enough to clear the noise, along the lines of the test designs we set out for AI citations. Until at least the first two are present, treat llms.txt as low-cost housekeeping.

Those three queries are the shape of the real demand. People are not asking how to write the file. They are asking whether it does anything, and whether Google will ignore it. That is a question about evidence, and it deserves an answer built from documentation rather than from optimism. The same discipline applies to the rest of AI visibility work, which is why measuring GEO beyond clicks and rankings and monitoring brand mentions in AI matter more than any single file. Engine behaviour also diverges: Perplexity and ChatGPT choose sources differently, so a single file could not be the lever for both even if one of them read it.

Frequently asked questions

Does ChatGPT read llms.txt?

There is no OpenAI documentation saying it does. OpenAI’s publisher documentation covers four crawlers, GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, and directs webmasters to robots.txt and published IP ranges to manage access. OpenAI does publish an llms.txt for its own developer documentation, which is the reverse arrangement: it is helping agents read OpenAI’s docs, not reading yours. If ChatGPT fetches your /llms.txt, it is because a user or a tool pointed it at the URL.

Does Google use llms.txt for AI Overviews or AI Mode?

No. Google Search Central’s documentation on AI features states that “there are no additional requirements to appear in AI Overviews or AI Mode”, and that “you don’t need to create new machine readable files, AI text files, or markup to appear in these features”. Eligibility rests on the page being indexed and able to show a snippet, on crawling being allowed in robots.txt, and on the content being useful. Chrome’s Lighthouse audit checking for the file is a separate, experimental product decision and carries no ranking weight.

Will publishing llms.txt hurt my site?

No, and that is a large part of why it keeps spreading. It is a static text file with no directive power, so it cannot block a crawler or change how a page is indexed. The only real costs are the time to write it and the risk of it going stale and pointing agents at pages that no longer exist. Serve it with a 200 and correct content type, or return a clean 404 if you do not want one.

What is the difference between llms.txt and robots.txt?

robots.txt controls access. It names a user agent and states what that crawler may fetch, and every major AI company has built its publisher opt-outs on it. llms.txt suggests attention. It assumes a crawler already has access and offers a curated reading order. One is enforced by convention across the whole crawling ecosystem, the other is a proposal with no confirmed consumer among the search engines. If you only do one, do robots.txt.

What changed in version 2 of the llms.txt spec?

Version 2 landed on 10 August 2026, the first revision since launch. It added standard ways for agents to find Markdown versions of individual pages: a URL pattern where the Markdown copy sits at the same address with .md appended or substituted, plus the rel=”alternate” type=”text/markdown” and rel=”describedby” link relations, usable in HTTP Link headers. It also downgraded the Optional section from a mechanical rule to a convention, and replaced the speculative examples with real agent usage. Existing files need only a small change.

Should a UK small business bother with llms.txt in 2026?

Only after the work that is measurable. If your pages are indexed, your entity details are consistent, your structured data is correct and your robots.txt says what you mean, then twenty minutes on a curated llms.txt is reasonable housekeeping for the agent traffic that may arrive later. If any of those four are unresolved, the file is a distraction. The term draws 1,600 UK searches a month at low competition, which tells you how much attention it is getting, and nothing at all about whether it works.

Do documentation platforms generate llms.txt automatically?

Several do, and that is where most of the files in the wild come from. Documentation hosting tools generate one from the page tree without the site owner deciding anything, which is why the count of published files is a poor proxy for adoption by AI engines. A file existing on thousands of domains tells you a platform shipped a feature. It does not tell you that anything reads it.

By Beata Nowak, Strategy Lead, AiBoost | Published 29 December 2025 | Updated 12 September 2026 | 13 min read