llms.txt — the honest lesson

ExpertDuration ~6 min video + 40 min hands-onTools A text editor, Your site’s content map, llmstxt_generator.py (provided)

Why this lesson

Every few months a new “must-do” file appears in the AI-search conversation, and right now it’s llms.txt. You’ll see confident threads claiming it’s how you get ChatGPT to cite you. You’ll also see equally confident posts calling it a hoax. Both can’t be right. This lesson gives you the truth an expert needs: what the file is, how to write one that actually follows the spec, and — the part most tutorials skip — the honest evidence on whether it does anything. The short version: it’s a genuinely useful idea, it’s cheap to ship, and as of 2026 there is no confirmation that any major AI system reads it. Knowing that difference is what separates you from the people selling it as magic.

Watch

Segment: ~1:40–3:00 — the llms.txt notewatch full video

Watch for: Sam Oh's honest note on llms.txt inside the technical-AEO lesson: it's a proposed standard like robots.txt, but as of now no major LLM provider officially supports it — OpenAI doesn't use it, Anthropic publishes one without confirming its crawlers read it, and Google hasn't adopted it. His verdict: it won't hurt, but don't prioritise it over robots.txt, which is still the file that matters most.

The explainer

What llms.txt actually is

llms.txt is a proposed convention: a single Markdown file you publish at yourdomain.com/llms.txt that gives an AI system a clean, curated map of your site. It was proposed by Jeremy Howard (of Answer.AI and fast.ai) in September 2024 at llmstxt.org. The problem it tries to solve is real: a normal web page is full of navigation, ads, cookie banners and scripts, and an AI has a limited context window. llms.txt is meant to hand the model the important stuff directly — who you are, what you cover, and links to your best pages — as plain, easily parsed Markdown. That’s why it’s sometimes called a markdown site file.

The spec has a strict, simple shape. Follow it exactly or tools will treat the file as malformed:

# Project or site name

> A one-line blockquote summary of what this site is and who it's for.

Optional free-form paragraph(s) giving more context. No headings here.

## Docs

- [Getting started](https://example.com/start): How to set up in five minutes.
- [API reference](https://example.com/api): Full endpoint documentation.

## Guides

- [SEO basics](https://example.com/seo): A plain-language intro to search.

## Optional

- [Changelog](https://example.com/changelog): Older release notes, safe to skip.

The rules that matter: exactly one H1 (#) with the name at the very top; an optional blockquote (>) summary; then any number of ## Section headings, each holding a bullet list where every item is a link in the form - [Title](url): Description. A section literally named ## Optional marks links an AI can skip if it’s short on context. There’s also llms-full.txt — a companion file at /llms-full.txt that inlines your entire documentation as one big Markdown document, so a model can ingest everything in a single fetch instead of following links.

The honest part

Here’s what the hype threads leave out. llms.txt is an unratified standard — a proposal one person published, not something agreed by the AI providers or any standards body. And adoption by the models that would need to read it has not happened. As Sam Oh puts it plainly: OpenAI doesn’t use it. Anthropic publishes one on its own site but hasn’t confirmed its crawlers actually read it. Google hasn’t adopted it — Google’s own John Mueller and Gary Illyes have said Google does not use llms.txt, with Mueller comparing it to the long-dead keywords meta tag. No major LLM provider has confirmed reading it, and independent trackers (Peec AI’s much-shared “helper or hoax?” post is the honest one to read) find the measured effect on citations is close to zero.

Adoption on the publishing side is growing — Anthropic, Perplexity, Stripe and Cloudflare all ship one — but publishing a file and having crawlers consume it are two different things. Contrast this with robots.txt, which every serious crawler genuinely obeys and which can silently block GPTBot or ClaudeBot from your whole site. That’s the file that actually decides whether AI sees you (lesson 3.3). llms.txt is the file people argue about on Twitter.

So the expert stance is neither “skip it” nor “it’s the secret.” It’s this: ship a spec-valid one, because it’s cheap and it won’t hurt, then spend your real effort on citability, crawler access, schema and brand mentions — the things with evidence behind them. When (or if) the providers adopt it, you’re already compliant. Until then, you don’t rely on it, and you don’t sell it to a client as a ranking lever.

Do it

1. Pick your practice site and list your 8–15 most important URLs — your home, key product/service pages, cornerstone guides, contact/about.
2. Write llms.txt by hand following the spec: one # H1 name, a one-line > blockquote summary, then 2–4 ## Section headings grouping your links as - [Title](url): Description.
3. Add a ## Optional section for links a model could safely skip (changelogs, archives).
4. Validate the shape: exactly one H1, blockquote directly under it, every bullet a proper Markdown link with a description. A single stray heading level makes it malformed.
5. Optionally generate llms-full.txt by concatenating your key pages as clean Markdown — or run the provided llmstxt_generator.py to build both files from a URL list automatically.
6. Publish at /llms.txt, then write one honest sentence in your workbook: why you shipped it and why you’re not counting on it.

Level 3 workbook — llms.txt spec checklist + the honest-impact scorecard (ships with llmstxt_generator.py)level-3-workbook.pdf113 KBOriginal course material — free to use

Terms introduced

Terms introduced in this lesson

• llms.txt
• llms-full.txt
• Markdown site file
• Unratified standard

Each term links to the glossary, where it is defined by the way you just used it.

Check

Check yourself

1. What is the current, honest status of llms.txt as of 2026?

   Google and OpenAI both officially require itIt has replaced robots.txt as the file that matters mostIt is banned by Google and will get your site penalizedIt is a proposed, unratified standard that no major LLM provider has confirmed it reads, with a measured citation effect close to zero

2. Which line belongs at the very top of a spec-valid llms.txt file?

   A single H1 with the site or project name (# Name)A block of JSON-LD schemaA robots.txt-style "User-agent: *" directiveA list of disallowed crawlers

3. Given that llms.txt is currently low-impact, what is the sensible way to treat it?

   Block all AI crawlers insteadShip a valid one because it is cheap and harmless, but prioritise the things that actually move AI visibility and never rely on llms.txt aloneSkip it entirely — it is a waste of timeSpend most of your GEO budget perfecting it

Check answers

You can move on when you can… write a spec-valid llms.txt from scratch and explain, honestly, why it’s currently low-impact — that no major LLM provider confirms reading it and its measured citation effect is roughly zero — so you ship one without relying on it.

Mark this lesson complete

Go deeper

• llmstxt.org — Jeremy Howard’s original proposal and the canonical spec. Read it, then read Peec AI’s “llms.txt: helper or hoax?” for the balanced skeptic’s view.
• Google Search Central — Mueller’s and Illyes’ public comments that Google does not use llms.txt. The primary-source counterweight to the hype.
• Next: 3.8 · Brand mentions & AI E-E-A-T — the off-site signals that do move AI visibility.