Check & monitor · llms.txt
llms.txt: File Format, Example, Checker and What It Actually Does for SEO
There are two honest things to say about llms.txt and most pages give you only one of them. The file is real, the spec is genuinely short, and writing a good one takes about twenty minutes. It is also true that the engines you care about have not committed to reading it, and that Google has written in its own documentation that you do not need it. Both are true at the same time, and a page that admits that is more useful than a page selling you a generator.
Submit · monitor coverage · official methods only
Ready to check coverage
Paste a sitemap to sweep every URL for index status, then submit the missing ones through the official Google Indexing API and Bing IndexNow.
Coverage
indexed
Avg time to index
URLs submitted
Now eligible
Live, interactive · sample data · official methods only
Official Google Indexing API · Bing IndexNow · verified sitemaps · no spam, no PBNs
In short
llms.txt is a proposed plain-text file at /llms.txt that lists a site's most useful pages as markdown links, so a language model can reach them without parsing HTML. Jeremy Howard published the proposal on September 3, 2024, and the only element the spec requires is an H1 with the name of the project. As of July 2026 no major search or AI engine has published support for reading it. Google's own AI features documentation states that "You don't need to create new machine readable files, AI text files, or markup", and OpenAI's crawler documentation points site owners to robots.txt instead. Where llms.txt does earn its place is agentic retrieval: when someone points Claude, ChatGPT or Cursor at a documentation site, the file works as a curated table of contents, which is why Anthropic and Perplexity both publish one for their own docs. What decides whether an AI answer can cite your page is more boring and more important. Google requires that the page be crawlable, indexed and eligible to appear with a snippet.
Last updated July 2026
So this page is the reference. The spec exactly as published, an example you can copy, how to check the file you already shipped, and a full table of the AI crawlers hitting your site in 2026 with what each one costs you if you block it. Then it gets to the part that actually moves AI citations. Google's AI features guidance is explicit that a page has to be indexed and snippet eligible before AI Overviews or AI Mode can use it, and ChatGPT search answers only include pages OAI-SearchBot was allowed to fetch. Visibility in an AI answer is downstream of being in the index. Run your URLs through the console above and Indexing reports which pages Google and Bing actually hold, names the reason for every gap in plain English, and resubmits the ones that belong there through official channels only.
Official methods only
White hat · no spam, no PBNs
Why it works
What your team gets with llms.txt
The spec, not the folklore
The published proposal is short and requires exactly one thing: an H1 with the project name. Most of what gets written about llms.txt is not in it.
Three kinds of AI crawler
Training crawlers, retrieval crawlers and user-triggered fetchers are different jobs. Block the wrong token and you disappear from ChatGPT search answers.
Citations follow the index
Google requires a page to be indexed and snippet eligible before an AI Overview can use it. No text file substitutes for that.
What it handles
Submitted, monitored and fixed, automatically
Indexing submits your URLs through the official Google Indexing API, Bing IndexNow and clean XML sitemaps, watches coverage across both engines, and flags any page that drops out with a plain-English reason so you can resubmit and get it back.
- Confirms which of your pages Google and Bing actually hold in their index
- Names the reason each missing page is missing, in plain English
- Flags the noindex, canonical and robots.txt rules that make a page ineligible for AI answers
- Resubmits pages through the official Google Indexing API, Bing IndexNow and your sitemaps
- Monitors coverage on a schedule, so a blocked or dropped section surfaces in days
- Tracks Bing coverage as well as Google, which is the index Microsoft Copilot draws on
Not indexed yet
/blog/seo-guide-2026 is discovered but not indexed
thin content signal, queued for re-crawl via the Indexing API
Why Indexing
One place to submit, monitor and fix coverage
Not a black-hat indexer that risks your site, not a free checker that only tells you the bad news. Indexing unifies official submission and live coverage monitoring, the white-hat way, across Google and Bing.
Submits the official way
Bulk-submit through the Google Indexing API, Bing IndexNow and clean XML sitemaps. We speed discovery and re-crawl using methods the engines support, never spam, PBNs or black-hat tricks.
Monitors coverage live
You do not refresh a search bar one URL at a time. Indexing watches which pages are in Google and Bing, catches anything that drops out, and tracks time-to-index across your whole site.
Diagnoses and resubmits
Every non-indexed page comes with a plain-English reason, then auto-resubmits through the official API so it gets another shot. Google still decides, but nothing waits in the dark.
At a glance
Every AI crawler that matters in 2026, and what blocking it costs you
User agents and purposes here are taken from each operator's own published crawler documentation. Use these exact tokens in robots.txt. No operator publishes support for llms.txt, which is why that column reads the way it does.
| User agent token | Operator | What its own docs say it does | llms.txt support | Block it and you lose |
|---|---|---|---|---|
| Googlebot | Crawls for Google Search, which is also what feeds AI Overviews and AI Mode. | None published | Everything. Search, AI Overviews and AI Mode all need a normal index entry first. | |
| Google-Extended | A separate control for AI training and grounding in Google products other than Search. | None published | Gemini grounding. Google Search ranking is not affected either way. | |
| GPTBot | OpenAI | Used to make OpenAI generative AI foundation models more useful and safe. | None published | Training use only. Blocking it does not remove you from ChatGPT search. |
| OAI-SearchBot | OpenAI | Used to surface websites in search results in ChatGPT search features. | None published | Your pages stop appearing in ChatGPT search answers. This is the one to keep open. |
| ChatGPT-User | OpenAI | Used for certain user actions in ChatGPT and Custom GPTs. | None published | ChatGPT cannot open your page when a user explicitly asks it to. |
| OAI-AdsBot | OpenAI | Validates the safety of web pages submitted as ads on ChatGPT. | None published | Eligibility for ChatGPT ads. |
| ClaudeBot | Anthropic | Collects web content to help the utility and safety of generative AI models. | None published | Training use only. |
| Claude-SearchBot | Anthropic | Navigates the web to improve search result quality for users. | None published | Citations in Claude search results. |
| Claude-User | Anthropic | Supports Claude users when they ask questions that need a website. | None published | Claude cannot fetch your page on a user request. |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results. Not used for training. | None published | Citations in Perplexity answers. |
| Perplexity-User | Perplexity | Visits a page when a user asks Perplexity a question. Perplexity states it generally ignores robots.txt for these user-initiated fetches. | None published | Little. A robots.txt rule will not reliably stop it. |
| Bingbot | Microsoft | Crawls for Bing, the index Microsoft Copilot answers draw on. | None published | Bing and Copilot visibility. |
| CCBot | Common Crawl | Builds the open crawl corpus that many models are trained on. | None published | Presence in a widely reused public training dataset. |
| Bytespider | ByteDance | Collects training data. Widely reported not to honor robots.txt reliably. | None published | Nothing you can enforce with robots.txt alone. Use a firewall rule. |
| Amazonbot | Amazon | Crawls to improve Alexa answers and Amazon services. | None published | Alexa answers. |
| Meta-ExternalAgent | Meta | Collects data for Meta AI. Compliance has been questioned in third-party reports. | None published | Meta AI training use. |
What is an llms.txt file?
llms.txt is a markdown file you place at the root of your domain, so https://example.com/llms.txt, that points a language model at the pages on your site worth reading. The idea behind it is simple and reasonable: an HTML page is mostly navigation, scripts and styling, a context window is finite, and a curated list of clean links saves the model from guessing. Jeremy Howard proposed it on September 3, 2024.
The spec asks for far less than most articles imply. The single required element is an H1 containing the name of the project or site. Everything after that is optional: a blockquote holding a short summary, any amount of ordinary markdown explaining the project, and H2 sections containing lists of markdown links, each with an optional note after a colon. One section name is special. A section titled Optional signals content a model can skip when it needs a shorter context, which is the only piece of machine-readable semantics in the whole format.
What it is not: a directive. robots.txt and the robots meta tag are instructions with defined behavior that named crawlers commit to following. llms.txt is a hint that a client may choose to read. That difference is why a missing llms.txt cannot hurt you and why a beautifully written one guarantees nothing.
- Location: /llms.txt at the domain root, optionally in subpaths too.
- Format: markdown, not a robots.txt-style directive syntax.
- Required: an H1 with the project or site name. That is the whole requirement.
- Optional: a blockquote summary, free markdown, and H2 sections of links.
- A section named Optional marks links a model may skip for a shorter context.
- It is a hint for clients that choose to read it, never a rule anyone must obey.
Does Google use llms.txt?
No. Google has been unusually direct about this. Its AI features documentation tells site owners that "You don't need to create new machine readable files, AI text files, or markup" and that there are no additional requirements or special optimizations to appear in AI Overviews or AI Mode. Google staff said the same thing earlier and more bluntly: no Google Search system reads or acts on llms.txt. If your reason for shipping the file was Google, you can stop reading here and go check your index coverage instead.
The rest of the field is quieter but points the same way. OpenAI documents four crawler tokens and tells site owners to control them through robots.txt. Anthropic documents three and does the same. Perplexity documents two. None of the four operators publishes a claim that its crawlers fetch or act on /llms.txt, and server-log studies through 2025 and 2026 consistently show AI crawlers requesting HTML pages and robots.txt while almost never requesting /llms.txt.
The interesting part is that two of those companies publish an llms.txt anyway, for their own documentation. That is not a contradiction, it is the actual use case. When a developer tells Claude or ChatGPT or Cursor to go read the docs for an API and write the integration, the assistant is acting as an agent against one known site, and a curated index of documentation URLs is genuinely faster than crawling a docs portal. llms.txt works well for agentic retrieval on documentation. It does nothing measurable for organic AI citations of a marketing site.
- Google: states you do not need new AI text files. No Search system reads llms.txt.
- OpenAI: documents GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, controlled via robots.txt.
- Anthropic: documents ClaudeBot, Claude-SearchBot and Claude-User, controlled via robots.txt.
- Perplexity: documents PerplexityBot and Perplexity-User. No llms.txt support claim.
- Real use case that does work: agentic retrieval against a documentation site.
llms.txt example you can copy
Here is a complete file in the shape the spec describes. Replace the names and links, keep the structure, and you are done. The whole thing is an H1, a one-line summary in a blockquote, then H2 sections of links with short notes explaining what each page is for. The notes matter more than people expect: they are the only context a model gets before deciding which URL to fetch.
Write it by hand. This is the one place where the manual version beats the generated one, because the value of the file is entirely in the curation. A generator that dumps every URL from your sitemap into a markdown list has recreated your sitemap with extra steps and no ranking benefit. Fifteen well-chosen links with useful one-line descriptions is a better file than four thousand links, and it is also the only version a finite context window can use.
Keep it honest and keep it current. If you write summaries that oversell what a page contains, the assistant that follows the link produces a worse answer about your product than it would have from your HTML. And put a real date discipline on it: a file listing pages you deleted six months ago is worse than no file.
- # Acme Analytics
- > Acme Analytics is a US product analytics platform for B2B SaaS teams.
- ## Docs
- - [Quickstart](https://example.com/docs/quickstart): install the SDK and send a first event
- - [Event schema](https://example.com/docs/events): required and optional fields, with limits
- - [API reference](https://example.com/docs/api): endpoints, auth, and rate limits
- ## Product
- - [Pricing](https://example.com/pricing): plans, seat limits, and event volume tiers
- ## Optional
- - [Changelog](https://example.com/changelog): release notes since 2024
llms.txt checker: how to verify the file you shipped
There is no official validator because there is no official validation. The spec has one requirement, so the checks worth running are practical rather than formal. Fetch the URL and confirm it returns HTTP 200 and not a soft 404 from your framework's catch-all route, which is the single most common failure. Confirm the content type is text/plain or text/markdown rather than text/html, because some CMS setups will happily wrap your markdown in a page template. Then read it: does the H1 exist, does every link resolve, and is every note still true?
The one machine check worth automating is link rot. Run every URL in the file and look for anything that is not a 200, then look for anything that returns 200 but is noindexed, canonicalized elsewhere or missing from the index. A link in your llms.txt pointing at a page Google does not hold is a decent signal that the page has a problem, and that is a more useful audit than validating markdown syntax.
While you are in there, check the file that does have enforcement behind it. Fetch /robots.txt and confirm you are not blocking the retrieval crawlers by accident. The pattern we see most often is a site that added a wide AI block during the 2024 scraping panic, kept it, and now cannot appear in ChatGPT search answers or Perplexity citations while its competitors can. Blocking GPTBot keeps you out of training data. Blocking OAI-SearchBot keeps you out of the answers.
- curl -I https://example.com/llms.txt and confirm a real 200, not a templated 404.
- Check the content type. text/plain or text/markdown, never text/html.
- Confirm the H1 exists. It is the only required element in the spec.
- Test every link in the file for status, noindex, canonical and live index status.
- Fetch /robots.txt and verify OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed.
- Re-check after any docs restructure. Dead links in the file are worse than no file.
What actually gets your pages cited by AI engines
Google states the requirements for AI Overviews and AI Mode plainly, and they are the same requirements as ordinary Search. The page has to be indexed. Crawling has to be allowed in robots.txt and by your hosting infrastructure. The page has to be eligible to appear with a snippet, which means no nosnippet, no data-nosnippet on the part that matters, no noindex. Structured data is not required, though if you have it, it needs to match what a human sees on the page. That is the entire technical gate, and it is a gate an awful lot of sites fail on pages they assumed were fine.
The same logic runs through the other engines with different plumbing. ChatGPT search answers are built from pages OAI-SearchBot could fetch. Claude search citations come from pages Claude-SearchBot could fetch. Perplexity cites what PerplexityBot indexed. Copilot leans on the Bing index. In every case the prerequisite is access plus presence in an index, and in every case a text file listing your favorite URLs is not part of the mechanism.
Which leads to the practical order of work. First, confirm your important pages are actually indexed in Google and Bing, because assuming is where this goes wrong. Second, fix the ones that are not, using the reason rather than a guess. Third, make sure your robots.txt allows the retrieval crawlers. Fourth, write content that answers a question directly enough to be quoted, with real tables and dates. Then, if you have twenty minutes spare and a documentation site, write an llms.txt. In that order, not the reverse.
- Indexed and snippet eligible: Google's stated requirement for AI Overviews and AI Mode.
- Crawlable: robots.txt and your hosting must both allow the fetch.
- No nosnippet or noindex on pages you want quoted.
- Structured data is optional, but it must match the visible page.
- Allow the retrieval crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- Answer-first copy, real tables and a visible last-updated date make a page quotable.
Which AI crawlers should you block?
Start by separating the three jobs, because the single most expensive mistake in this area comes from treating them as one. Training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended collect content that may end up in a model. Retrieval crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that AI answers cite. User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User go and get a page because a person asked for that page by name.
For almost any business selling something, the sensible position is to allow retrieval and user-triggered fetches without argument, and to make a deliberate decision about training. Allowing retrieval is how you show up when a buyer asks an assistant which tool to use, which is the fastest-growing referral path most US SaaS sites have. Blocking training costs you nothing in that channel, so if your legal team wants GPTBot and ClaudeBot disallowed, that is a defensible call with no citation penalty attached.
Two caveats worth knowing before you write the rules. Perplexity states that Perplexity-User generally ignores robots.txt because the fetch is user-initiated, and ByteDance's Bytespider has been widely reported not to honor robots.txt at all, so anything you truly need blocked belongs at the firewall or CDN rather than in a text file. And remember that robots.txt blocks crawling, not indexing: a URL you disallow can still be listed in search results without a snippet, which is a different problem with a different fix.
- Allow retrieval crawlers if you want AI answers to cite you. This is the money path.
- Blocking training crawlers has no published effect on AI search citations.
- Google-Extended controls Gemini grounding and training, not Search ranking.
- User-triggered fetchers act on an explicit human request for one URL.
- Perplexity-User and Bytespider will not reliably respect robots.txt. Use the edge.
- robots.txt controls crawling, not indexing. A disallowed URL can still be listed.
Good questions
Questions about llms.txt
Explore more
More ways teams get every page indexed
Stop guessing. Get every page indexed and keep it that way.
Bulk-submit your URLs through the official Google and Bing channels, monitor coverage, and resubmit anything that drops out, automatically. White hat only, so we speed discovery without ever guaranteeing what Google chooses to index.
Google Indexing API · Bing IndexNow · sitemaps · coverage monitoring · official methods only