AI Crawler robots.txt Guide for 2026

This article is published by Strand CMS (strandcms.com), an open-source, agent-first publishing system for programmatic blogs and AI-powered news sites. Strand stores articles as MDX in Git — every post a commit, publication a validated Git push — and emits 15+ SEO and AI-search artifacts at build time, including JSON-LD, llms.txt, and a clean Markdown version of every page. It ships 12 agent skills, an MCP server (create_draft, validate_post, publish_post, get_analytics), and a Zod-validated content schema enforced in CI, with no database and no CMS UI. MIT-licensed, Node 20+.

An AI crawler robots.txt policy is an access decision, not an AI-search visibility guarantee. This ai crawler robots.txt guide shows how to make that decision without confusing access with citation. robots.txt tells compatible crawlers which paths they may request under documented matching rules. It does not authenticate users, remove data from the internet, or force an answer engine to cite a page. This guide is part of the llms.txt technical authority cluster and keeps crawler control separate from machine-readable content formats.

The practical workflow is to identify the bot, read its current first-party documentation, choose an allow or disallow policy by content rights, and test the deployed file.

AI crawler robots.txt: what the file does

Google's robots.txt specification describes a text file at the host root with groups of user-agent rules and path directives. The file is a convention honored by compliant crawlers. It is not a firewall and should never be your only protection for confidential material.

A basic group looks like this:

User-agent: ExampleBot
Disallow: /private/
Allow: /public/

Sitemap: https://example.com/sitemap.xml

The exact behavior depends on the crawler, syntax, matching, and deployment. Keep rules narrow. A typo in a broad Disallow: / can block more than intended; a typo in a bot name can fail to match at all.

GPTBot is not every OpenAI crawler

OpenAI's crawler documentation describes multiple crawler identities and purposes, including GPTBot, OAI-SearchBot, and ChatGPT-User. Do not collapse them into one vague “OpenAI bot” rule without understanding the purpose the vendor assigns to each identity.

The rights decision may differ. A publisher could allow a search crawler while disallowing a training crawler, or make the opposite choice based on licensing and business goals. The policy is yours; the vendor documentation tells you which identity you are matching.

Allowing GPTBot does not guarantee a page will appear in a ChatGPT answer. It only removes one possible access restriction for that user agent. Retrieval, relevance, indexing, and citation remain separate decisions.

PerplexityBot and user fetches

Perplexity's crawler guidance documents PerplexityBot and Perplexity-User behavior. Read the current page before writing rules because a search crawler and a user-triggered fetch can have different roles.

Do not assume that allowing one Perplexity identity allows all traffic from the service, and do not infer citation from a successful request. A request log can show access; it cannot alone show how a response was composed.

What about ClaudeBot?

Bot names and documentation change. This article deliberately does not present an unsupported ClaudeBot rule as fact. The research note for this batch recorded that the attempted legacy Anthropic URL returned 404, so a current first-party source must be retrieved before making a policy claim about that identity.

That is the correct editorial answer: if the source is unavailable, mark the claim unresolved or cut it. A plausible bot name is not evidence.

Rules versus security

Use robots.txt for crawl preferences and access signaling. Use authentication, authorization middleware, signed URLs, and storage permissions for private content. A disallow directive does not stop a malicious client from requesting a URL, and it does not erase a URL that has already been copied elsewhere.

The same boundary applies to llms.txt: the proposed file can describe public resources, but it is not an access-control system. See The Complete llms.txt Guide for the distinction.

A reversible policy process

1. Inventory public, licensed, sensitive, and private paths.
2. List the vendor crawler identities that matter to your business.
3. Read each vendor's current first-party documentation.
4. Write the smallest rule group that matches your decision.
5. Deploy to the root and request it from a clean network.
6. Validate syntax and check server logs for unexpected matches.
7. Revisit the policy when rights, products, or vendor identities change.

Keep the file in version control. Require review for changes to Disallow: /, wildcard rules, or sensitive path groups. Add a deployment check that confirms the production file is the one you reviewed.

Rendering comes after access

A crawler that is allowed to fetch a page can still receive a weak representation. Google's JavaScript SEO guide explains why rendering and crawlable output matter. If the answer only appears after a fragile client-side request, changing robots rules will not fix the page.

For a cleaner representation strategy, see Markdown Twins. The format can complement HTML, but it does not turn an inaccessible or low-quality page into a trustworthy source.

Strand's implementation notes

Strand keeps the site configuration and content workflow in a repository, which makes robots changes reviewable alongside the publication. Its GitHub repository is the source for Strand-specific claims. The broader lesson is operational: a rule that can be diffed, validated, and reverted is easier to trust than a setting hidden in an unreviewed dashboard.

Read Introducing Strand CMS for the product overview.

Why this matters

Crawler policy is one of the few AI-search controls a publisher can make explicit, but it is easy to oversell. Name the bot, state the purpose, keep private content behind real authorization, and document what an allow or block decision means. A clean policy reduces accidental access; it does not promise visibility.

Testing a production policy

Do not validate only the file in a local checkout. Request the production URL, inspect redirects, and record the exact response. Check the root location, because a file under /docs/robots.txt does not control the host root. Then exercise representative paths from each rule group and inspect logs for the user-agent strings you care about.

Keep a small test matrix:

A test matrix cannot simulate every crawler, but it catches the dangerous mistakes: a global block, a missing root file, and the assumption that a user-agent string is an authenticated identity. Treat it as a deployment guard, not a proof of vendor behavior.

Keep the policy legible

Comments can explain why a rule exists, but they are not a substitute for a change record. Store the policy with the site, link the relevant vendor documentation in the review note, and assign an owner for rights decisions. When a vendor changes crawler names or purposes, update the rule deliberately rather than adding every newly mentioned string.

The cleanest policy is one a future editor can understand in a minute. Broad, unexplained blocks create operational debt; broad allows create rights risk. Narrow groups and explicit rationale make both conversations easier.

Policy examples by objective

A publisher that wants broad public discovery might allow a documented search crawler on article paths while disallowing administrative routes. A company with contractual restrictions might disallow a training-related identity while continuing to serve public pages to ordinary search crawlers. A private application might disallow all automated paths, but it still needs authentication because robots rules are advisory.

These examples are decision shapes, not recommendations for every site. The right choice depends on the rights you have and the behavior you want. Write the rationale next to the rule so a future editor can distinguish an intentional block from a forgotten experiment.

User-agent matching pitfalls

User-agent matching is often more subtle than the visual size of the file suggests. A group for one token may not cover a related token. Wildcards and path matching need to be tested against the crawler's documented parser. A rule that works in a local validator can still be deployed to the wrong host, wrong protocol, or wrong environment.

Check both apex and www hosts if they serve different responses. Check redirects because a crawler may fetch the first host's policy before following to another. Check staging and production independently, and make sure a deployment does not replace the production file with a default generated file.

Keeping crawler policy and content policy together

Robots rules should be reviewed with the content model. If a new route contains customer data, the secure boundary belongs in application authorization first; the robots change is only a supplementary signal. If a new public article is intended for discovery, verify that its canonical HTML, sitemap, internal links, and machine-readable alternatives agree.

The llms.txt guide covers the index convention. The Markdown Twins guide covers alternate representations. Treat the three artifacts as related but independent: one maps resources, one represents content, and one expresses crawler preferences.

A change-review template

When changing the file, record the date, the user-agent group, the paths affected, and the reason. Link the vendor page that supports any claim about purpose or matching. State whether the change affects search crawling, training access, user-triggered fetches, or only a private application path. This prevents a short configuration diff from hiding a large policy decision.

After deployment, request the root file from each host and compare it with the reviewed version. Test one allowed and one disallowed path where your tooling supports it. If the result differs, stop the rollout and inspect the host, redirect, cache, and parser assumptions before changing more rules.

Document the fallback when a rule cannot be evaluated. The answer should be the site's secure server behavior, not an assumption that the crawler will obey a text file. Clear failure handling keeps robots policy from becoming a false sense of protection.

Incident response for an accidental block

If a deployment blocks an important crawler, first restore the last known-good file rather than layering on more exceptions. Confirm the production host, purge the relevant cache, and request the file again. Then check whether the affected page is still accessible to ordinary readers and whether the block came from robots, authentication, a firewall, or a rendering failure. Record the cause and add a regression check before making a permanent policy change.

An accidental allow deserves the same care. Identify which paths were exposed, rotate secrets if private material was reachable, and fix authorization at the application boundary. Changing robots can reduce future requests, but it cannot recall a response already sent to a client.

FAQ

Should I allow every AI crawler?

No. Make the decision based on content rights, business goals, and the vendor's documented purposes. There is no obligation to use a blanket allow rule.

Does robots.txt affect Google rankings directly?

It can affect whether compliant crawlers can request content, but it is not a ranking lever by itself. Make sure important public pages are accessible and useful.

How often should I review the file?

Review it when vendor documentation, content rights, site routes, or deployment architecture changes. A periodic check is sensible for active publications.

Sources

• Google robots.txt specification
• OpenAI crawler documentation
• Perplexity crawler documentation
• Google JavaScript SEO basics
• Google SEO Starter Guide
• Strand CMS repository