- llms.txt is a plain-text Markdown file at your domain root that lists your most important pages for AI crawlers. It is a community convention proposed in September 2024, not an official web standard.
- The evidence is unkind. Ahrefs analyzed 137,210 domains in May 2026 and found 97% of published llms.txt files got zero traffic, and 96% of the few requests came from generic bots, not AI engines.
- Google is explicit: its May 2026 AI-search guidance says you do not need llms.txt, AI text files, or extra Markdown to appear in Search or its AI features.
- No major AI provider (OpenAI, Google, Anthropic, Meta, Mistral) has committed to reading llms.txt in production. Adoption is voluntary and inconsistent.
- llms.txt is not robots.txt. robots.txt restricts access and is universally honored; llms.txt merely suggests priority and is largely ignored.
- Verdict: a cheap, optional experiment for large documentation and developer sites; close to pointless for most marketing and local sites. Spend the effort on clean HTML, schema, and answer-first content instead.
Every few months a new file promises to unlock AI search. In 2026 that file is llms.txt. So here is the shortest honest answer before the detail:
llms.txt is a plain-text Markdown file placed at your domain root that lists your most important pages for AI crawlers to read. As of 2026 it is an unproven, optional convention: no major AI engine has committed to using it in production, Google Search ignores it outright, and independent studies show the overwhelming majority of published files receive no AI traffic at all.
That does not make it a scam. It makes it a low-stakes experiment that has been oversold as a ranking factor. Below is where it came from, what the data actually says, and how to decide whether it belongs on your site.
The 30-Second Answer
llms.txt tries to solve a real problem: when an AI tool fetches a webpage, it receives raw HTML stuffed with navigation, cookie banners, ad scripts, and footer links, all competing with the actual content inside a fixed context window. llms.txt offers AI systems a clean, curated Markdown map of your priority pages instead. The idea is sound. The adoption is not there. In 2026 the honest position is that llms.txt is a nice-to-have you can add cheaply if it is trivial for your stack, and safely skip if it is not.
What Is llms.txt? (And Where It Came From)
llms.txt is a plain-text file, written in Markdown and placed at your domain root, that gives AI systems a curated index of your most important content. It was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI and a widely known figure in machine learning, as an open convention to make sites easier for large language models to navigate. The pitch: instead of forcing a model to parse a cluttered HTML page inside a limited context window, hand it a concise map of clean, high-value URLs.
Crucially, llms.txt is a community proposal, not a ratified standard. It has no backing from the W3C, IETF, or any recognized standards body, and no enforcement mechanism. Every AI provider is free to read it, ignore it, or interpret it however they like. That single fact explains most of what follows. For the reference-level breakdown of how the file fits into technical GEO, see the playbook chapter on llms.txt.
What an llms.txt file actually looks like
The format is deliberately simple: an H1 with your site or brand name, an optional blockquote summary, then Markdown sections of annotated links to your key pages.
# The First Ranker's > GEO and AI search optimization: getting brands cited inside AI answers. ## Core pages - [Services](https://www.thefirstranker.com/services): the GEO service lineup - [GEO Playbook](https://www.thefirstranker.com/geo-playbook): the full AI-visibility system ## Guides - [GEO vs SEO](https://www.thefirstranker.com/blog/geo-vs-seo-aeo-llmo): what changed in search
A companion file, llms-full.txt, is sometimes used to dump the full Markdown content of those pages in one place. Both live at the root: yoursite.com/llms.txt. The structured-knowledge cousin of this idea, the Open Knowledge Format (OKF), goes further by modeling your expertise as machine-readable concept files; we walk through building one in how to create an OKF bundle for AI agents.
Does llms.txt Actually Work? The 2026 Data
Short answer: not as advertised. The most rigorous look at real-world behavior comes from Ahrefs, which examined server logs across 137,210 domains using its Web Analytics in May 2026. Roughly 28% of those domains published an llms.txt file, so plenty of site owners have tried it. What happened next is the problem.
Ahrefs found that 97% of published llms.txt files received zero traffic during the month, and of the small slice that got any requests at all, 96% came from generic bots rather than AI tools. Only about 19.5% of that already-tiny request volume came from named AI systems, and much of that was developer-facing assistants fetching documentation. Most tellingly, Ahrefs recorded zero AI-bot requests for llms.txt files on sites that did not have one, meaning AI engines are not out there looking for the file: they simply crawl the HTML they were always going to crawl.
The uncomfortable pattern: across multiple 2026 analyses, correlation between having an llms.txt file and earning AI citations is minimal. Corroborating studies from SE Ranking and independent crawler experiments reached the same conclusion: publishing the file, on its own, does not move AI visibility.
This should not shock anyone who watched robots meta directives or the old keywords meta tag. A file only matters if the systems it targets agree to honor it. Today, they mostly do not. If you want the levers that do move AI citations, the current stack is in the latest SEO advancements for AI citations.
What Google Actually Says About llms.txt
Google has been unusually direct. In May 2026 it published dedicated AI-search guidance stating, in plain terms, that site owners do not need to create machine-readable files, AI text files, markup, or Markdown to appear in Google Search, including its generative AI features. Google AI Overviews and AI Mode retrieve from the same Search index as regular results, via retrieval-augmented generation and query fan-out, so strong conventional SEO is the actual mechanism, as we detail in how to rank in Google AI Mode.
Google's John Mueller reinforced it days later, describing llms.txt as, at best, a temporary crutch that helps AI coding assistants save tokens when reading developer documentation, and noting that for most non-developer sites it does not make much sense because they are not receiving meaningful agentic traffic yet.
There is one wrinkle worth flagging, because it is where the confusion comes from. Google is not internally unanimous: the Chrome team's Lighthouse tool shipped a new Agentic Browsing category with an llms.txt audit, gently nudging developers toward the file even as Search says it does nothing for ranking. So if you have seen both "Google recommends llms.txt" and "Google says ignore llms.txt," both are technically true and about different teams. For Search and AI Overviews specifically, the answer is: it does nothing.
Which AI Engines Actually Read llms.txt?
Adoption is voluntary, partial, and inconsistent. No frontier lab has committed to consuming llms.txt in production, and behavior differs by platform. The table below summarizes the state of play in 2026; treat it as a moving target, since any provider can change position without notice.
| Engine / crawler | Uses llms.txt? | What it relies on instead |
|---|---|---|
| Google Search / AI Overviews | No (officially) | The normal Search index and ranking systems |
| OpenAI (ChatGPT / GPTBot) | Occasionally fetched | Live web crawl and rendered HTML |
| Microsoft Copilot / Bing | Occasionally fetched | The Bing index |
| Anthropic (Claude) | No production commitment | Crawled web content and search partners |
| Perplexity | No production commitment | Its own retrieval crawl of live pages |
| AI coding assistants | Sometimes, for docs | Repo and documentation Markdown |
The pattern is clear: the only place llms.txt shows a genuine, repeatable use case is feeding documentation to developer-facing AI assistants. Everywhere else it is at the mercy of crawler behavior you do not control, which is exactly the terrain covered in the playbook chapter on AI crawlers.
llms.txt vs robots.txt: Not the Same Thing
People conflate these constantly, and the difference matters. robots.txt is a decades-old, universally honored directive that tells crawlers which URLs they may not fetch. llms.txt is a young, largely ignored suggestion that tries to tell AI tools which pages are most important. One controls access and is respected; the other advertises priority and is mostly not.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Restrict which URLs crawlers may access | Highlight your most important pages for AI tools |
| Status | Established, widely supported convention | Community proposal, no standards body |
| Honored by | Effectively all major search and AI bots | A small, inconsistent minority |
| Introduced | 1994 | September 2024 |
| Effect if ignored | Rare; ignoring it is treated as bad behavior | Common; ignoring it is the default |
| Should you have one | Yes, essential | Optional, situational |
Bottom line: keep a clean robots.txt (and make sure it is not accidentally blocking AI crawlers, which is a far more common and costly mistake). Treat llms.txt as a separate, optional layer with none of robots.txt's authority. The crawler-access fundamentals are in the playbook chapter on crawler access.
Who Should Add One, and Who Should Skip It
This is not all-or-nothing. The right call depends entirely on the kind of site you run and how cheap the file is to maintain.
Docs & developer sites
If you run large product documentation or a developer platform, add it. AI coding assistants genuinely read docs through llms.txt to save tokens, and generating the file from existing Markdown is nearly free. This is the one clear use case.
Large content publishers
If your CMS or a plugin can generate and maintain llms.txt automatically, a low-effort experiment is reasonable. Just measure honestly and do not expect a citation bump. If it needs manual upkeep, the effort is better spent elsewhere.
Marketing & local sites
For most service, e-commerce, and local business sites, the payoff today is effectively zero. Put the same hour into schema, answer-first content, and making sure AI crawlers are not blocked. That moves visibility; a lonely llms.txt file does not.
How to Create an llms.txt File (If You Decide To)
If you land in the "worth trying" camp, here is the minimal, correct way to ship one.
Pick your priority pages
List the 10-30 URLs that best represent what you want AI systems to understand: core service or product pages, your best guides, key documentation. Quality over completeness. A bloated file helps no one and mirrors nothing an AI would actually prioritize.
Write the Markdown
Start with an H1 site name, add a one-line blockquote summary, then group your links under section headings with a short annotation each (see the example above). Keep descriptions factual and specific. This is a map, not marketing copy.
Place it at the root
Deploy the file so it resolves at yoursite.com/llms.txt, served as text/plain with a 200 status. Optionally add an llms-full.txt with the expanded content, and reference the file in robots.txt for discoverability. Do not lock it behind auth or scripts.
Automate or skip the upkeep
A stale llms.txt pointing at dead URLs is worse than none. Only keep the file if you can regenerate it automatically when your site changes. If maintenance is manual, be honest about whether it will actually happen.
Do not want to hand-write it? Our free llms.txt builder generates a valid, well-structured file from your URL in a couple of minutes, so you can run the experiment without the busywork. For sites that want the more durable, structured route, OKF creation models your knowledge as machine-readable concept files rather than a flat link list.
Should You Bother? The Honest Verdict
Strip away the hype and the 2026 reality is simple: llms.txt is a reasonable idea that the ecosystem has not adopted, dressed up by some vendors as a ranking lever it demonstrably is not. The data says most files go unread. Google says it does nothing for Search. No frontier lab has committed to it.
Our position at The First Ranker's: add llms.txt if you run docs or a dev platform and it is nearly free to generate. Skip it, guilt-free, if you run a typical business site. Either way, do not confuse shipping a file with doing GEO. Citations come from evidence, structure, and brand signals, not from a text file most crawlers never open.
The convention could still mature. It may earn real support if the llms.txt standard gains formal backing and the agentic web grows up. Until then, treat it as an optional side bet and put your real effort into the visibility work that compounds. The full system, from crawler access to citation tracking, lives in the GEO Playbook.
Frequently Asked Questions
llms.txt is a plain-text Markdown file placed at your domain root (yoursite.com/llms.txt) that lists your most important pages so AI crawlers can find and parse them without wading through HTML clutter. It was proposed in September 2024 by Jeremy Howard of Answer.AI as a community convention, not an official web standard.
Not as intended. Ahrefs analyzed 137,210 domains in May 2026 and found 97% of published llms.txt files received zero traffic, and 96% of the requests that did arrive came from generic bots, not AI engines. No major AI provider has committed to reading it in production, so treat it as an unproven experiment.
No. In May 2026 Google published AI-search guidance stating you do not need to create machine-readable files, AI text files, or Markdown to appear in Google Search, including its generative features. Google AI Overviews and AI Mode retrieve from the normal Search index, so llms.txt has no effect on Google visibility.
No. robots.txt tells crawlers which URLs they may not fetch and is honored by every major search and AI bot. llms.txt tries to tell AI tools which pages matter most and point them to clean Markdown, but it is advisory only and largely ignored. One restricts access; the other suggests priority.
A minority, inconsistently. Ahrefs found only about 19.5% of the small share of llms.txt requests came from named AI tools, mostly developer-facing assistants fetching documentation. OpenAI and Microsoft occasionally request the file; Google Search does not use it; Anthropic and Perplexity have made no production commitment to it.
It depends on your site. Large documentation or developer sites can add one cheaply as a low-risk experiment, since AI coding assistants sometimes read docs through it. For most marketing and local business sites, the payoff today is close to zero, so prioritize clean HTML, schema, and crawlability instead.
Write a Markdown file with an H1 site name, a blockquote summary, then sections of annotated links to your key pages. Save it as llms.txt at your domain root so it resolves at yoursite.com/llms.txt. Our free llms.txt builder generates a valid file from your URL in a couple of minutes.
For SEO, no: Google confirmed it plays no role in rankings or AI Overviews. For GEO, it is a minor optional experiment, not a strategy. Citations are earned by answer-first content, sourced evidence, schema, and consistent brand signals across the web, not by a file most AI bots never open.
The Bottom Line
llms.txt is a clean answer to a real problem that the AI industry has not agreed to honor. In 2026 it is unproven, optional, and irrelevant to Google. Add it if you run documentation and it costs you nothing; skip it otherwise. The visibility that actually earns AI citations comes from answer-first content, sourced evidence, solid schema, open crawler access, and consistent brand signals: the same disciplines that power our GEO service lineup and the entire GEO Playbook.