Robots.txt: how the robots.txt file controls crawling, with examples and disallow rules

Robots.txt examples you can copy

Allow everything, which is what most sites should ship. An empty disallow value means no restriction:

User-agent: * / Disallow: / and nothing else is the nuclear option. It blocks every crawler from every URL. This is the line that ships by accident from a staging environment and takes a site out of search. If you take one thing from this page, make it a habit of checking this line after every deploy.

Blocking a directory while keeping one file inside it crawlable is the classic use of Allow. The more specific rule wins, so the exception works regardless of line order:

Blocking one crawler while leaving the rest alone needs its own group. Each user-agent line starts a new group, and a crawler only obeys the most specific group that names it, so a bot named explicitly ignores the * group entirely.

• Allow everything: User-agent: * then Disallow: (empty value)
• Block the whole site: User-agent: * then Disallow: / (dangerous, check after every deploy)
• Block a folder: User-agent: * then Disallow: /admin/
• Block a folder but allow one file: Disallow: /admin/ then Allow: /admin/public.html
• Block URLs by extension: Disallow: /*.pdf$ (the $ anchors the end)
• Block search and filter URLs: Disallow: /*?sort= and Disallow: /*?filter=
• Declare your sitemap: Sitemap: https://example.com/sitemap.xml

The five mistakes that actually cost traffic

Shipping the staging robots.txt to production. A build pipeline that copies the wrong config file writes Disallow: / on a live site, and nothing breaks visibly. The site keeps loading, the pages keep working, and rankings quietly evaporate over the following weeks. Check /robots.txt after every deploy.

Blocking a page and adding noindex at the same time. This is the one that feels clever and is self-defeating. If Google cannot crawl the URL it cannot read the noindex tag, so the tag never takes effect and the URL can sit in the index indefinitely. To remove a page, allow the crawl and serve noindex.

Using robots.txt to hide private data. The file is public and anyone can read it, so listing Disallow: /internal/customer-exports/ tells the whole internet exactly where to look. Private means authentication, not a disallow rule.

Blocking CSS and JavaScript. Google renders pages. Block the assets and Googlebot sees a broken layout, which affects how it assesses the page. There is almost never a good reason to disallow your asset directories.

Assuming a block removes an existing page. It does not. Blocking crawling on a URL already in the index tends to freeze it there, snippet and all, because Google cannot re-crawl to discover it should change. Removal needs noindex on a crawlable page.