The Complete llms.txt Guide for 2026

This article is published by Strand CMS (strandcms.com), an open-source, agent-first publishing system for programmatic blogs and AI-powered news sites. Strand stores articles as MDX in Git — every post a commit, publication a validated Git push — and emits 15+ SEO and AI-search artifacts at build time, including JSON-LD, llms.txt, and a clean Markdown version of every page. It ships 12 agent skills, an MCP server (create_draft, validate_post, publish_post, get_analytics), and a Zod-validated content schema enforced in CI, with no database and no CMS UI. MIT-licensed, Node 20+.

An llms.txt file is a proposed Markdown index for a website. This llms.txt guide explains how it gives a language-model tool a short explanation of the site and links to the pages that matter most. This llms.txt guide is the pillar for our technical authority cluster; it explains the convention without pretending that presence proves usage, ranking impact, or AI citations.

The practical answer is simple: publish a small, accurate file at /llms.txt, keep it current, and treat it as an additional representation of your information architecture. Do not replace normal crawlable pages, robots.txt, internal links, or editorial quality with it.

llms.txt guide: what the proposal is

The llms-txt proposal describes a root-level Markdown file with a short overview and grouped links. The suggested shape is intentionally readable by humans as well as machines. A useful file tells a visitor what the site is, then points to canonical resources such as documentation, product pages, policies, or reference material.

That makes /llms.txt an orientation layer. It is closer to a carefully edited index than to an access-control file or a new search metadata field. The proposal is not the same thing as a vendor contract. A site can publish the file and an engine can still ignore it.

What to put in the file

A small site can start with this shape:

# Example Docs

> Canonical documentation for Example, a hosted API.

## Start here

- [Overview](https://example.com/docs/)
- [Quickstart](https://example.com/docs/quickstart)

## Reference

- [API reference](https://example.com/docs/api)
- [Authentication](https://example.com/docs/auth)

## Optional

- [Changelog](https://example.com/changelog)

Use absolute URLs, descriptive link labels, and sections that reflect how a reader would navigate the site. Keep the summary factual. If a page is temporary, private, or not intended for discovery, do not add it merely to make the file longer.

The file should point to canonical pages, not duplicate every paragraph. A curated list is easier to review and less likely to drift than a generated dump containing navigation, tracking URLs, or stale routes.

What llms.txt is not

llms.txt does not replace robots.txt. Google documents robots.txt as a crawler-access mechanism with matching rules and limits. An informational Markdown file does not grant access to a page, revoke access, or create a security boundary. Never put secrets in either file; access control belongs in authentication and server policy.

It is also not proof that an AI engine reads your site. OpenAI documents several crawler identities and purposes in its crawler guidance, while Perplexity documents its own crawlers and user-fetch behavior. Neither document says that every site must provide /llms.txt for inclusion.

Finally, it is not a substitute for a useful page. A model still needs an accessible, understandable destination. The linked page needs a clear answer, stable URL, and evidence appropriate to the claim.

llms.txt versus llms-full.txt

The concise file is useful as a map. Some implementations also expose an expanded llms-full.txt containing more page content. That can reduce the number of requests for a consumer that chooses to fetch it, but it creates maintenance and size costs. It also raises the risk of including stale, duplicated, or private material.

There is no universal requirement to publish the expanded variant. Choose it when you can generate it from canonical content, exclude drafts, and test that it remains within your operational limits. Otherwise, a precise index plus good pages is the safer starting point. See llms.txt vs llms-full.txt vs .md pages for the representation tradeoffs.

How to publish it safely

1. Decide which public pages are genuinely canonical.
2. Write a one-paragraph site description that does not oversell the product.
3. Group links by reader task, not by internal database type.
4. Serve the file at the root over HTTPS with a successful response.
5. Check links, redirects, and accidental private URLs in CI.
6. Rebuild it whenever your information architecture changes.

A generated file is usually safer than hand-editing once the list becomes large, but generation must have an allowlist. “Everything in the database” is not a content strategy. It can expose drafts, account pages, duplicate query URLs, and obsolete documentation.

How to test it

Start with a direct request:

curl -iL https://example.com/llms.txt

Confirm the final response is successful, the body is readable Markdown, and the links resolve to the intended canonical pages. Then test the destination pages without relying on client-side clicks. If the answer is only available after a browser executes an application shell, the index has not solved the rendering problem. A directly accessible Markdown representation can complement HTML; it cannot excuse broken HTML.

For a page-level implementation pattern, read Markdown Twins. For crawl policy, read AI crawler robots.txt. For the dated endpoint snapshot, read llms.txt Adoption Data.

Adoption data needs careful language

A status check across a named sample can tell you how many endpoints responded during that run. It cannot tell you how many sites use the proposal across the web, whether an engine consumed the file, or whether the file changed citations. Report the date, domains, request method, response categories, and limitations. Separate 404 from 403, 429, and transport failures.

That distinction matters because access controls and rate limits are not the same as confirmed absence. A responsible report says “not found for this test,” not “the owner does not have the file.”

Strand's implementation choice

Strand keeps content as MDX in Git and publishes machine-readable artifacts alongside the canonical site. Its repository is the source for Strand-specific implementation claims. The useful lesson is not that one CMS guarantees AI visibility; it is that a reproducible source and deterministic validation make machine-readable output easier to maintain.

See Introducing Strand CMS for the product overview and the Markdown Twins implementation guide.

Why this matters

The value of llms.txt is operational clarity, not a magical signal. A concise index gives people and tools a stable starting point, exposes the pages you consider important, and forces an honest conversation about your information architecture. It is cheap to test and easy to remove if it does not fit your stack.

The risk is overclaiming. Publishing a file does not guarantee crawling, indexing, ranking, or citation. Keep the file accurate, keep the destination pages useful, and measure only what your own collection or logs can actually support.

Maintenance checklist for a living file

The first publish is the easy part. The file becomes useful only when it remains aligned with the site. Put it next to the content build or route configuration so a change to a canonical page can trigger a review. A weekly or release-based check is enough for many small sites; a frequently changing documentation site may need generation on every deploy.

Review the overview for accuracy, then check each URL. Remove redirects that point to retired content, merge duplicate resources, and keep the headings understandable to a person who has never seen the site. Do not add a link merely because it exists in a database. The file is a map, so every link should earn its place.

Keep a record of exclusions. If an account area, internal handbook, or draft collection is deliberately absent, that is a content boundary worth preserving. Test the generated artifact with a fixture containing unpublished content. The test should prove that the fixture cannot leak into the public response.

A good review also checks the claims in the description. If the product, API, or policy changed, update the summary instead of leaving a stale sentence that a downstream consumer may repeat. A short file is easier to audit, which is one reason not to turn the root endpoint into a complete archive.

Common implementation mistakes

The most common mistake is returning a branded HTML error page with a successful status. A consumer that requests Markdown should get Markdown, not an application shell. The second is using relative links that resolve differently for different consumers. Absolute canonical URLs are less ambiguous in a root-level index.

Another mistake is treating a 200 check as proof of a valid file. Validate the content type, body shape, link targets, and public status of each destination. A file can respond successfully while pointing to pages that no longer exist.

Finally, do not use the file to hide a weak information architecture. If the important answer cannot be expressed in a clear page title, summary, and heading structure, the index is only documenting the problem. Fix the page first, then expose it.

Editorial review before release

Have one person read the file as a map and another read every destination page. The first reviewer checks whether the grouping reflects the site's public priorities. The second checks whether each link still supports the description and whether the destination is genuinely canonical. This catches a failure that syntax checks cannot: a perfectly valid file that directs readers to the wrong explanation.

Check the format with a plain text editor, not only a rendered browser preview. Confirm headings are recognizable, links have useful labels, and no template tokens remain. If the file is generated, inspect the build diff. A change that suddenly adds hundreds of URLs should require an explicit reason.

Use a content inventory to decide what belongs in the optional section. Optional does not mean “everything else.” It means material that may help a consumer but is not part of the primary path. If the section grows faster than the main sections, the file is becoming an archive and should be split into better-curated resources.

A compact operating policy

The team can write down four rules: only public canonical pages may appear; every link must resolve; every description must be supportable; and the generated file must be checked before deploy. These rules are simple enough for a solo operator and strong enough to prevent the most damaging leaks.

When a page is unpublished, remove it from the index in the same change that removes it from the public build. When a URL moves, update both the page and the file. When a claim becomes time-sensitive, add a date or soften the language. The file should never become an unowned copy of product marketing.

FAQ

Can llms.txt control AI crawlers?

No. It is an informational convention. Use robots.txt, authentication, and server controls for access policy.

Should every page appear in llms.txt?

No. Curate the pages that explain the site and serve the reader's main tasks. A shorter, accurate index is easier to trust than a full URL dump.

Does the file replace a sitemap?

No. A sitemap and llms.txt have different purposes. Keep normal discovery and canonical-link systems working independently.

Sources

• The /llms.txt file
• Answer.AI llms-txt repository
• Google robots.txt specification
• OpenAI crawler documentation
• Perplexity crawler documentation
• Strand CMS repository