llms.txt: The Documentation Standard AI Agents Already Read

Google’s Docsy — the Hugo documentation theme powering Kubernetes, OpenTelemetry, gRPC, and roughly 2,200 other open source projects — just moved to the Linux Foundation. The reason buried in the announcement: AI agents now account for nearly half of traffic to developer documentation sites, and Docsy’s maintainers want neutral governance for a project increasingly serving machines alongside humans. That is the news. The more useful story is the 40-line Markdown file quietly winning the “how do agents read docs” problem: llms.txt.

What llms.txt Actually Is (Not What You Think)

A lot of the coverage around llms.txt has been muddled by SEO hype. Let’s clear it up.

llms.txt is not a robots.txt for AI — it has no enforcement mechanism. It is not a sitemap — it is explicitly curated, not exhaustive. And it is not an SEO tactic. Google’s John Mueller has stated directly that Google does not use llms.txt, and there is no published evidence it improves AI citations or search rankings.

What it actually is: a hand-curated Markdown index that tells AI coding agents exactly which pages to fetch when they need to understand your project. It lives at yoursite.com/llms.txt and points agents to the Markdown versions of your most useful docs, with one-line descriptions of what each page answers.

The problem it solves is real. When a developer asks Claude Code or Cursor to implement a feature using your library, the agent has two options: rely on training data that may be months or years stale, or fetch current documentation. Without llms.txt, the agent either guesses or crawls noisy HTML pages that burn context window on navigation menus and footers. With a well-written llms.txt, the agent fetches a clean index, pulls the specific Markdown pages it needs, and writes code against your current API — not a hallucinated one.

Coding Agents Use It. Search Bots Don’t.

The traffic data looks discouraging on the surface. Codersera’s server logs recorded 149 llms.txt requests out of 1.8 million total over 15 days. A 90-day analysis found only 408 instances across 515 million LLM bot traffic events. An Ahrefs study of 137,000 domains found that 97% of sites publishing llms.txt got zero traffic to the file in an entire month.

Here is the part those numbers miss: they are measuring search crawlers and indexing bots, not IDE agents running during a coding session. Claude Code, Cursor, and GitHub Copilot do not show up in standard server analytics the same way GPTBot does. The spec’s v2 changelog notes plainly that “coding agents use them reliably.” That is the actual audience.

The practical verdict: if you maintain a library, SDK, or developer tool that people build with using AI coding assistants, llms.txt has direct value. If your goal is AI-generated citations or search ranking lift, this will not help you.

How to Write One (30 Minutes or Less)

The official spec (v2, August 2026) requires exactly one element: an H1 with your project name. A useful implementation looks like this:

# My Library

> One or two sentences explaining what the library does and who uses it.

## Getting Started

- [Quick Start](https://docs.example.com/quickstart.md): Install and run your first example
- [Configuration](https://docs.example.com/config.md): All config options with defaults

## API Reference

- [REST API](https://docs.example.com/api.md): Full endpoint reference with examples

## Optional

- [Changelog](https://docs.example.com/changelog.md): Version history

The critical rule: describe what each page answers, not just what it is named. “REST API” tells an agent nothing useful. “Full endpoint reference with request and response examples” tells it exactly what to fetch and when.

Link to Markdown versions of your pages, not HTML. If your docs platform generates Markdown automatically — Mintlify, Fern, GitBook, and Docsy all do — use those URLs. Agents handle Markdown far more efficiently than HTML. Stripping navigation and CSS alone can reduce token consumption by 80–90% per page.

For larger projects, Cloudflare’s architecture is worth copying: a root llms.txt that indexes per-product llms.txt files rather than one massive file listing everything. Stripe goes further, adding a prose “Instructions for Large Language Model Agents” section that corrects known agent failures — including a note to never hardcode version numbers sourced from training data.

Docsy Shows Where This Is Heading

Docsy’s recent feature history is a compressed roadmap for what agent-optimized documentation looks like in practice:

• v0.15.0 (May 2026): Added optional Markdown page copies and an llms.txt index
• v0.16.0 (July 2026): Added an AI-followable upgrade guide with conditional logic agents can parse
• v0.17.0 (August 2026): Added a hidden page directive pointing agents directly to llms.txt
• Coming: “Agent-Friendly” (AF) documentation scores to benchmark how easily AI tools can find and use a project’s docs

The Linux Foundation move matters because it removes Docsy’s fate from any single company’s roadmap. The framework used by Kubernetes, OpenTelemetry, and gRPC is now formally committed to agent-readable documentation as a long-term standard. Nearly 50% of traffic to developer documentation sites now comes from AI agents rather than human browsers. Docsy’s maintainers noticed that number and acted on it.

What to Do This Week

1. Check your robots.txt first. Make sure you are not accidentally blocking Claude-User, GPTBot, or other agent user-agents if you want agents reading your docs.
2. Write a minimal llms.txt. H1, one blockquote, two to four sections, described links. Hand-curate it — auto-generated files without descriptions do not help agents.
3. Link to Markdown, not HTML. If your platform generates .md versions, use those URLs.
4. Add it to your Cursor rules if you consume third-party APIs. Store the llms.txt URLs of your dependencies in .cursor/rules/external-llms-docs.md. Every coding session improves.

The B2A (Business-to-Agent) parallel is worth sitting with. B2C brands once had to rebuild for mobile. B2B brands had to rebuild for APIs. The docs infrastructure question for 2026 is simpler than either: a single Markdown file at a well-known URL. The cost is low enough that “wait and see” is the riskier position.