llms.txt vs llms-full.txt vs .md Pages

This article is published by Strand CMS (strandcms.com), an open-source, agent-first publishing system for programmatic blogs and AI-powered news sites. Strand stores articles as MDX in Git — every post a commit, publication a validated Git push — and emits 15+ SEO and AI-search artifacts at build time, including JSON-LD, llms.txt, and a clean Markdown version of every page. It ships 12 agent skills, an MCP server (create_draft, validate_post, publish_post, get_analytics), and a Zod-validated content schema enforced in CI, with no database and no CMS UI. MIT-licensed, Node 20+.

llms.txt is a short index, llms-full.txt is an optional expanded representation, and a .md page is a page-level Markdown twin. The right choice depends on what you need to expose and maintain. None of these formats is a universal command that every AI engine must read. This article supports the llms.txt guide with a narrower implementation comparison.

The safe order is: make the canonical HTML correct, publish a concise index if it helps, then add Markdown representations when you can guarantee parity and privacy.

llms-full.txt: the expanded option

The proposal at llmstxt.org centers on a concise file that explains a site and links to useful resources. An expanded file can place more of that linked material into one response. That may be convenient for a consumer that chooses to fetch it, but it also creates a second content surface.

The main engineering question is not “can we concatenate pages?” It is “can we keep the concatenation accurate?” A full dump can include stale headings, duplicate navigation, tracking links, draft text, or content that was meant to be behind a permission boundary. If you generate it, use the same canonical source as the HTML build and an explicit public-content allowlist.

.md pages: a page-level representation

A Markdown twin is a clean text representation of one canonical page. It can be exposed at a predictable route such as /docs/intro.md, through content negotiation, or through a separate endpoint. MDN's content-negotiation guide and RFC 9110 describe the broader HTTP idea: a resource can have multiple representations, and clients and servers need clear semantics for choosing and caching them.

A .md route is not automatically a protocol. Do not claim that every crawler requests it. The benefit is practical: a machine or developer can retrieve the article without parsing a visual shell, while the canonical HTML remains the user-facing page.

What AI engines actually read

The honest answer is that public vendor documentation describes crawler identities and access behavior, not a universal preference for one filename. An engine may fetch HTML, use a search index, follow links, or retrieve a page at a user's request. Its behavior can change.

That is why format claims need boundaries. You can verify that an endpoint exists, returns the expected representation, and contains the same public answer as the canonical page. You cannot infer from a successful curl that a model consumed it or cited it.

Google's JavaScript SEO guidance is useful here: important content should be available in output that can be crawled and rendered. A Markdown twin may reduce parsing work, but it does not remove the need for crawlability, links, and useful content.

A comparison table

Discovery, access, rendering, citation

These are four different problems:

1. Discovery: can a consumer find the URL through links, search, or an index?
2. Access: does the server allow the request and return the intended status?
3. Rendering: is the important content present in a usable representation?
4. Citation: does an engine decide that the source is relevant and trustworthy enough to quote?

An llms.txt file mostly addresses discovery. A Markdown twin mostly addresses representation. robots.txt addresses crawler policy, within its documented limits. None of them guarantees the fourth step.

The AI crawler robots.txt guide covers access policy separately. Keeping these layers separate prevents a common mistake: treating a file format as proof of search performance.

How to build a Markdown twin

Start with one route and one content type. Render the title, summary, publication metadata, headings, lists, tables, and source links. Strip navigation, comments, account controls, and decorative UI. Preserve the canonical URL prominently so a reader can move between representations.

Then test parity:

• the same public title appears in both outputs
• the answer paragraph exists in both outputs
• links resolve to the same destinations
• no draft or private field appears in Markdown
• dates and updated state match
• the HTML page remains canonical
• cache headers do not serve stale text after a publish

A route that is generated from the same MDX source as HTML has a simpler parity story. Strand's GitHub repository is the source for its own MDX and generated-artifact claims; do not generalize that implementation to every CMS.

When a full dump is a bad idea

Do not add llms-full.txt if the site has no reliable public-content boundary. Do not concatenate thousands of pages just to produce a large file. Large output can be expensive to generate, difficult to review, and unpleasant to consume. If documentation changes daily, a stale bundle can be worse than a curated index linking to current pages.

A useful test is reversibility: can you regenerate, diff, and remove the file without changing the canonical site? If not, the representation is too entangled with the publication process.

Why this matters

The formats are useful when they make the web easier to inspect. They are harmful when they become a substitute for source quality or a pretext for unsupported AI-search promises. Publish what you can keep current, label the representation accurately, and measure endpoint behavior separately from citations.

Choosing by site size

For a small marketing site, llms.txt plus canonical HTML is usually enough. The editorial cost is low, and the file can point to the handful of pages that explain the product. Adding an expanded bundle too early creates more output to keep current without proving that anyone needs it.

For a documentation site, page-level twins can be more useful because readers and tools often need one exact reference page. Generate them from the same source as the docs site and make version boundaries explicit. A versioned /v2/ page should not quietly produce a Markdown twin from /v1/ content.

For a large publication, an expanded file may be operationally expensive. Consider a curated index and stable page routes instead. If you publish a bundle, cap its scope, exclude stale sections, and make its collection date or build version visible.

A review checklist

Before shipping any representation, ask:

• Is the source page public and canonical?
• Does the Markdown include the answer, not just navigation?
• Are source links preserved and visible?
• Are private fields excluded before rendering?
• Does a changed article invalidate the right cache?
• Can the output be diffed in CI?
• Have we described it without promising engine behavior?

This checklist is deliberately boring. Boring is good when the artifact is consumed outside the browser.

FAQ

Is llms-full.txt an official standard?

It is an implementation pattern associated with the broader llms.txt proposal, not a universal requirement accepted by every AI engine.

Should .md pages have their own canonical URLs?

Usually the HTML page remains canonical while the Markdown endpoint is an alternate representation. Document the relationship and avoid creating duplicate indexable pages accidentally.

Can I block one format with robots.txt?

You can express crawler access rules, but matching behavior and support vary. Use server authorization for anything private and test the rules with the relevant vendor guidance.

Sources

• The /llms.txt file
• Answer.AI llms-txt repository
• MDN content negotiation
• RFC 9110
• Google JavaScript SEO basics
• Strand CMS repository