Guides · Guide
llms.txt and AI Crawler Access
Decide whether to add llms.txt, how it differs from robots.txt, and which OpenAI and Google crawlers actually control access.
The operator task is not “add llms.txt because AI search exists.” The task is to decide which access problem you actually have, then pick the file or crawler token that matches that problem.
llms.txt is a community Markdown map. Robots.txt is the access file for compliant crawlers. OpenAI documents separate bots for search visibility, training collection, and user-triggered fetches. Google documents crawler classes, robots rules, and fetch limits. Those layers do not collapse into one “AI crawler” switch.
This guide stays on access. It does not claim that publishing llms.txt will raise citations, rankings, or AI visibility. Google’s published crawler and AI-search guidance does not treat the file as a ranking or citation control. Do not add it as if it were one.
Decide the job before the file
Write the job in one sentence before you touch a file.
| Job | Control that matches | File that does not match |
|---|---|---|
| Keep a crawler off a path | robots.txt for that crawler’s token | llms.txt |
| Keep a page in Google’s crawl | Allow the URL, keep it crawlable, put useful HTML early | A Markdown map of other URLs |
| Stay out of ChatGPT search features | OAI-SearchBot rules | GPTBot rules |
| Ask not to be used for foundation-model training | GPTBot rules | OAI-SearchBot rules |
| Point agents at a short curated page list | Optional llms.txt, if you will maintain it | The full sitemap copied into Markdown |
| Make a page worth citing | Clear answer, evidence, dates, entities | Any access file |
If you cannot name the job, do not publish a new file. Access work that starts from a trend name usually blocks the wrong bot or advertises pages that are still thin.
Three files, three jobs
Keep these files separate.
| File | Job | Success looks like |
|---|---|---|
| robots.txt | Tell compliant crawlers which paths they may request | Public source pages are allowed; internal and junk paths are not |
| sitemap | List public URLs you want discovered | The list matches live, canonical, indexable pages |
| llms.txt | Offer a curated Markdown map for agents | A human editor would hand the same short list to a new analyst |
robots.txt is not a security boundary. A sitemap is not a quality score. llms.txt is not a robots replacement. A site can have all three, or only the first two. The missing file is not automatically a GEO gap.
OpenAI bots are not one crawler
OpenAI’s public crawler documentation splits the work. Recheck that page before you change tokens. The jobs below are documented there as of 2026-09-09 and re-checked on 2026-10-26, when the four-bot split was unchanged.
| Bot | Documented job | Operator meaning |
|---|---|---|
| OAI-SearchBot | Used to surface sites in ChatGPT search features | This is the search-visibility token. Opting out means the site will not be shown in ChatGPT search answers. Navigational links may still appear. |
| GPTBot | Crawls content that may be used in training generative AI foundation models | This is the training-collection token. Disallowing it is a training preference, not a ChatGPT-search switch. |
| ChatGPT-User | Used for some user-initiated visits | This is a user-triggered fetch. robots.txt may not apply. It is not the control for Search appearance. |
| OAI-AdsBot | Visits pages submitted as ads | Ad landing-page checks. Not documented as foundation-model training. |
Search robots.txt changes can take about a day to take effect. Do not treat a same-hour log gap as proof that the rule failed. One more current-page detail matters for log analysis: when a site allows both OAI-SearchBot and GPTBot, OpenAI may serve both use cases from a single crawl, and robots.txt fetches can carry an extra robots.txt marker in the user-agent string. Fewer log lines does not mean a bot stopped working.
The common mistake is to block GPTBot and then report “we opted out of ChatGPT.” That only stated a training preference. ChatGPT search visibility is a different token. User-triggered fetches can still happen after both rules are set.
Google access is still crawl and fetch
Google’s common crawlers, including Googlebot, respect robots.txt for automatic crawls. User-triggered Google fetchers are a different class. Identify Google crawlers by user-agent, source IP, and reverse DNS. Do not invent extra product names that are not on Google’s crawler overview.
Google also documents fetch size. Googlebot’s 2026-03-31 guidance says it fetches up to 2MB per URL, including HTTP headers, except PDFs at 64MB. Bytes after the cutoff are not fetched, rendered, or indexed. The first 2MB is treated as the complete file. Put titles, answers, evidence, and important links high in the HTML. Heavy inline assets can push useful text past the cutoff.
That is an access problem even when robots.txt allows the URL. A page can be permitted and still fail crawlability because the answer sits below a large blob of unused markup. See the dated note on Googlebot fetch limits and crawler IP paths.
llms.txt does not change those limits. Google has not published the file as a ranking lever, a citation lever, or a substitute for robots.txt.
When llms.txt is worth adding
Add the file when all of these are true:
- You already have a small set of public pages that answer real buyer tasks.
- A person on the team can name those pages without opening a keyword sheet.
- Those URLs are allowed in robots.txt and return useful HTML.
- Someone will update the file when a URL moves or a page is retired.
- You can describe the file as a map, not as an AI-ranking tactic.
Skip the file when:
- the site is still a thin template farm;
- the list would be a dump of the sitemap;
- no owner will maintain it;
- the real problem is a blocked path, a noindex, or a page that cannot be cited.
A short, dated, curated list can help an agent that chooses to read it. A long, stale, slogan-filled list is just another neglected URL.
Operator checklist
Work the list in this order. Stop when the job is done.
- Name the answer surface you care about. Google Search AI features, ChatGPT search, and training collection are different contests.
- Inventory the live robots.txt tokens. Do not collapse GPTBot, OAI-SearchBot, and Googlebot into one row.
- Confirm public source pages are allowed and reachable. Check status codes, not only the file text.
- Confirm the HTML answer is early enough to survive a truncated fetch.
- Decide training versus search versus user-triggered fetch as separate choices.
- Only then decide whether a curated llms.txt is worth maintaining.
- Recheck the official OpenAI bot page and Google crawler docs before you change a token. Do not copy a third-party snippet as if it were the spec.
Example
A documentation site wants ChatGPT search to keep using its public guides, does not want those guides in foundation-model training, and wants agents to find three evergreen pages.
The matching set is: allow OAI-SearchBot on public guides, disallow GPTBot on those paths if that is the training choice, leave Googlebot allowed, keep the answers in the first screen of HTML, and publish a short llms.txt that lists the three guides. The contact page stays out of the map.
Counterexample
A team publishes /llms.txt, copies every sitemap URL into it, blocks User-agent: *, and then asks why “AI SEO” did not move. The map points at pages the crawler cannot fetch. The sitemap and the Markdown file now describe a site that robots.txt has closed. The next edit is the access file, not another heading in llms.txt.
What this page refuses
- Treating llms.txt as a Google citation or ranking control.
- Treating GPTBot, OAI-SearchBot, and ChatGPT-User as one ChatGPT crawler.
- Using robots.txt as security for private material.
- Claiming a third-party adoption percentage as an operating constant.
- Linking access work to a single AI visibility score.
If the page can be fetched and the answer is still unused, move to content citability and how AI citations work. Access gets a URL into the pool. It does not finish the citation contest.