In this guide: exactly what robots.txt, sitemap.xml, llms.txt, and llms-full.txt each do for AI crawlers in 2026 — which bots read which file, whether you need all of them, and the copy-paste-ready configuration to ship.
Quick answer
They are not competitors. robots.txt controls access, sitemap.xml lists URLs, and llms.txt summarizes meaning. A complete 2026 setup ships all three.
See ours live: robots.txt, sitemap.xml, llms.txt, and llms-full.txt.
Three small text files sit at the root of every well-configured website, and in 2026 a fourth is joining them. They are constantly confused for one another — but each does a genuinely different job for the crawlers and AI engines reading your site.
This guide settles the confusion: what each file is for, which AI bots read which, and the exact configuration to ship.
TL;DR: Three Files, Three Jobs
| File | Job | Format | Primary audience |
|---|
robots.txt | Access control — who may fetch what | Plain text directives | All crawlers |
sitemap.xml | Discovery — list of all URLs | XML | Search engines |
llms.txt | Meaning — curated summary | Markdown | AI engines |
llms-full.txt | Corpus — full structured catalog | Markdown | AI ingest / RAG |
robots.txt — Access Control
robots.txt is the oldest of the four (1994) and answers one question: which crawlers are allowed to fetch which paths? It is a list of User-agent blocks with Allow / Disallow rules, plus — by convention — a Sitemap: line.
It does not tell crawlers what your content means, and it does not list your pages. It is a bouncer, not a tour guide. Critically, it is also the discovery hub: because nearly every crawler fetches robots.txt first, it is where you advertise the location of your other files.
sitemap.xml — URL Discovery
sitemap.xml answers “what URLs exist and when did they change?” It is a machine-exhaustive XML list of every canonical URL with optional lastmod, changefreq, and priority hints. Search engines use it to discover and prioritize crawling.
It carries no summary and no meaning — just locations and timestamps. If your site is large or has pages that are not well-linked, a sitemap is how crawlers find them all. Need one? Our free sitemap generator builds a valid one, and the sitemap checker validates it. For why this matters specifically for chatbot training, see our sitemap-for-chatbot-training guide.
llms.txt — Meaning & Summary
llms.txt is the new file, and it answers a question the other two never could: “what is this site about, and what matters most?” It is a curated Markdown document at your root — a one-paragraph identity, your key sections, and links to your most important pages, written for AI consumption.
Where a sitemap dumps every URL with no context, llms.txt is selective and meaningful. It is the difference between handing someone a phone book and handing them a one-page brief on who to actually call. Proposed in 2024 at llmstxt.org, it is increasingly fetched by AI engines as a discovery and summary hint.
llms-full.txt — The Full Corpus
llms-full.txt is llms.txt’s comprehensive companion. Same Markdown format and root location, but bigger: instead of a curated index, it inlines structured metadata for every page worth ingesting — titles, URLs, dates, descriptions, keywords. An AI agent that fetches it can understand most of your site without crawling page by page.
The division of labor: llms.txt is the dust jacket, llms-full.txt is the full index at the back of the book. Ship the small one by hand; generate the big one from your content catalog so it stays current. We walk through exactly how we auto-generate ours in how we built our llms-full.txt.
Need a Clean Sitemap First?
Generate and validate a crawler-ready sitemap.xml in seconds with our free tools — the foundation AI crawlers build on. No signup required.
Which AI Crawlers Read What
The major AI crawlers identify themselves with documented user-agent strings and respect robots.txt:
- GPTBot (OpenAI) — trains and fetches for ChatGPT; honors robots.txt.
- OAI-SearchBot / ChatGPT-User (OpenAI) — browse-time fetches.
- ClaudeBot (Anthropic) — trains and fetches for Claude.
- PerplexityBot (Perplexity) — indexes for citation-first answers.
- Google-Extended — controls use of your content for Gemini / AI training, separate from regular Googlebot.
- CCBot (Common Crawl) — feeds many downstream AI datasets.
These bots all read robots.txt for access and sitemap.xml for discovery. The llms.txt / llms-full.txt convention is newer and adoption is uneven — but the cost of publishing is so low, and the downside zero, that there is no reason to wait for universal support.
Do You Need All Three (or Four)?
Short answer: yes, with effort scaled to your site.
- robots.txt — always. Even an “allow everything” file with a Sitemap line is correct and useful.
- sitemap.xml — always for any site past a handful of pages.
- llms.txt — yes; it is an afternoon of work and the AEO upside is real.
- llms-full.txt — yes if you have a content library (blog, docs, tools). For tiny sites, llms.txt alone covers the same ground.
The Recommended 2026 Config
Put this in robots.txt so one fetch reveals every other file:
User-agent: *
Allow: /
# Sitemaps
Sitemap: https://yourdomain.com/sitemap.xml
# AI / LLM ingest files (llmstxt.org convention)
# https://yourdomain.com/llms.txt
# https://yourdomain.com/llms-full.txt
Then ship a curated /llms.txt (identity, key sections, top pages, short FAQ) and, if you have a content library, an auto-generated /llms-full.txt.
Allowing & Blocking AI Bots
If you want to allow everything (the right default for most marketing sites), the wildcard above is enough. If you want to block specific AI crawlers from training on your content while still allowing search engines, target them by name:
# Block AI training crawlers but keep search visibility
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Be deliberate here. Blocking these bots protects content but forfeits AI citations and the referral traffic that comes with them. For a blog or docs site whose goal is reach, blocking is usually the wrong call — you are opting out of the fastest-growing discovery channel. Decide per content type.
Common Setup Questions
- Can I have multiple sitemaps? Yes — list each with its own
Sitemap: line, or use a sitemap index file. - Does llms.txt need to list every page? No — that is what llms-full.txt (or your sitemap) is for. Keep llms.txt curated.
- What content type should llms.txt be served as?
text/plain; charset=utf-8. It is Markdown, but plain-text content type is correct. - Will blocking in robots.txt remove me from AI answers? Over time, for compliant bots, largely yes — that is the point of blocking.
Next Steps
Get the foundation right, then layer on meaning:
- Generate and validate your sitemap with our free generator and checker.
- Add the robots.txt config above, advertising all your files.
- Publish a curated llms.txt; auto-generate llms-full.txt if you have a content library.
- Read the AEO guide and GEO Playbook to turn this plumbing into actual citations.
These four files are the technical foundation of AI visibility. They take an afternoon to get right and then quietly work for years.
What is the difference between llms.txt, robots.txt, and sitemap.xml?
They do three different jobs. robots.txt is an access-control file that tells crawlers which paths they may or may not fetch. sitemap.xml is a discovery file that lists your URLs (with last-modified dates) so crawlers find everything. llms.txt is a meaning file — a Markdown summary that tells AI systems what your site is about and points them at your most important content. robots.txt says "you may enter here", sitemap.xml says "here is everything", and llms.txt says "here is what matters and what it means".
Do AI crawlers actually respect robots.txt?
The major, named AI crawlers do. OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot, Google-Extended, and Common Crawl's CCBot all document the user-agent strings they use and honor robots.txt directives for them. As with traditional crawling, robots.txt is a voluntary standard — a badly behaved scraper can ignore it — but the reputable AI companies that drive citation traffic do comply, so it is the correct place to express your access policy.
Is llms.txt an official standard?
Not a formal W3C standard yet. llms.txt is a 2024 proposal from Jeremy Howard / Answer.AI, documented at llmstxt.org, and adoption is growing organically rather than through a standards body. Several AI tools and open-source RAG frameworks already prefer it when present. The pragmatic read: it is where robots.txt was in the mid-1990s — not formally mandated, increasingly respected in practice, and cheap enough to ship that waiting for official blessing makes little sense.
Does llms.txt replace my sitemap.xml?
No — they serve different consumers and different purposes. sitemap.xml is a machine-exhaustive list of URLs primarily for search-engine crawlers; it has no summary or meaning, just locations and timestamps. llms.txt is a curated, human-readable Markdown summary for AI systems; it is selective, not exhaustive. Keep your sitemap.xml for Google and Bing, and add llms.txt for AI engines. They coexist at your site root and target different audiences.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the lightweight, hand-curated index — typically 5-10 KB summarizing your site and linking key pages. llms-full.txt is the comprehensive companion — a larger machine-generated file (often 80-200 KB) that inlines structured metadata for every page worth ingesting. The pattern most teams use: write llms.txt by hand and refresh it when pricing or top features change; generate llms-full.txt automatically from your content catalog so it never goes stale. We document our own implementation in a dedicated build write-up.
Should I block AI crawlers in robots.txt?
It is a genuine trade-off, not an obvious yes or no. Blocking GPTBot, ClaudeBot, and others protects your content from being ingested — appropriate for paywalled, proprietary, or sensitive material. But it also forfeits AI citations and the referral traffic and brand authority that come with them. For most marketing sites, blogs, and documentation, allowing AI crawlers is the better call: the upside of being cited in AI answers outweighs the downside of being ingested. Decide per content type, not site-wide by reflex.
Where do these three files live and how do AI crawlers find them?
All three live at your site root: yourdomain.com/robots.txt, yourdomain.com/sitemap.xml, and yourdomain.com/llms.txt (plus yourdomain.com/llms-full.txt). Crawlers check robots.txt first by convention, and your robots.txt should contain a Sitemap: line pointing to your sitemap and comment lines advertising your llms.txt and llms-full.txt. That way a crawler that fetches robots.txt — which essentially all of them do — discovers the location of every other file from that single entry point.
How do I generate llms-full.txt without maintaining it by hand?
Generate it from your existing content catalog at build time. On Next.js, add a route handler at app/llms-full.txt/route.ts marked force-static that imports your blog and tool metadata and emits Markdown — it prerenders once per build with no runtime cost. Static-site generators (Astro, Hugo, Eleventy) use a template that loops content collections; WordPress can write the file on publish via a PHP snippet. The point is that the file derives from your real content, so new pages appear automatically and it never drifts out of date.
