Robots.txt file: disallow rules, noindex, and how to block Googlebot without losing pages

Robots.txt examples you can copy

Allow everything, which is what most sites should ship. An empty disallow value means no restriction:

User-agent: * / Disallow: / and nothing else is the nuclear option. It blocks every crawler from every URL. This is the line that ships by accident from a staging environment and takes a site out of search. If you take one thing from this page, make it a habit of checking this line after every deploy.

Blocking a directory while keeping one file inside it crawlable is the classic use of Allow. The more specific rule wins, so the exception works regardless of line order:

Blocking one crawler while leaving the rest alone needs its own group. Each user-agent line starts a new group, and a crawler only obeys the most specific group that names it, so a bot named explicitly ignores the * group entirely.

• Allow everything: User-agent: * then Disallow: (empty value)
• Block the whole site: User-agent: * then Disallow: / (dangerous, check after every deploy)
• Block a folder: User-agent: * then Disallow: /admin/
• Block a folder but allow one file: Disallow: /admin/ then Allow: /admin/public.html
• Block URLs by extension: Disallow: /*.pdf$ (the $ anchors the end)
• Block search and filter URLs: Disallow: /*?sort= and Disallow: /*?filter=
• Declare your sitemap: Sitemap: https://example.com/sitemap.xml

The five mistakes that actually cost traffic

Shipping the staging robots.txt to production. A build pipeline that copies the wrong config file writes Disallow: / on a live site, and nothing breaks visibly. The site keeps loading, the pages keep working, and rankings quietly evaporate over the following weeks. Check /robots.txt after every deploy.

Blocking a page and adding noindex at the same time. This is the one that feels clever and is self-defeating. If Google cannot crawl the URL it cannot read the noindex tag, so the tag never takes effect and the URL can sit in the index indefinitely. To remove a page, allow the crawl and serve noindex.

Using robots.txt to hide private data. The file is public and anyone can read it, so listing Disallow: /internal/customer-exports/ tells the whole internet exactly where to look. Private means authentication, not a disallow rule.

Blocking CSS and JavaScript. Google renders pages. Block the assets and Googlebot sees a broken layout, which affects how it assesses the page. There is almost never a good reason to disallow your asset directories.

Assuming a block removes an existing page. It does not. Blocking crawling on a URL already in the index tends to freeze it there, snippet and all, because Google cannot re-crawl to discover it should change. Removal needs noindex on a crawlable page.

robots.txt for AI crawlers: the tokens that matter in 2026

The most consequential robots.txt decisions being made right now are about AI crawlers, and a lot of sites are making them by accident. During 2024 plenty of teams added a broad block against anything that looked like an AI bot. Some of those rules are still in place, and the sites carrying them are now invisible in exactly the place their buyers are starting to look. robots.txt is the file that decides this, because every major operator documents robots.txt as the control surface and none of them uses any other file.

The distinction that matters is between training and retrieval, because they are different crawlers with different tokens. OpenAI documents GPTBot for training its foundation models and OAI-SearchBot for surfacing sites in ChatGPT search results, and it says that a site opting out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic documents ClaudeBot for training and Claude-SearchBot for search quality. Perplexity documents PerplexityBot for surfacing and linking sites in its results. Google keeps them separate too: Google-Extended controls AI training and grounding in Google products other than Search, and toggling it has no effect on Search ranking.

So a policy of blocking training while allowing retrieval is entirely writable, and for most businesses selling something it is the right one. Two limits are worth knowing. User-initiated fetchers behave differently: Perplexity states that Perplexity-User generally ignores robots.txt because a person asked for that specific page, and ByteDance's Bytespider has been widely reported not to honor the file at all, so anything you genuinely need enforced belongs at the CDN or firewall. And the usual rule still applies here: disallowing a crawler stops the fetch, it does not remove anything from an index.

• Training tokens: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended.
• Retrieval tokens that decide AI citations: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
• User-triggered tokens: ChatGPT-User, Claude-User, Perplexity-User.
• Google-Extended affects Gemini training and grounding, never Search ranking.
• Google requires a page to be crawlable and indexed before AI Overviews can use it.
• Audit any AI block written in 2024. It may be costing you ChatGPT and Perplexity citations today.