Robots.txt vs Noindex: Which Keeps a Page Out of Google?

robots.txt vs noindex: which actually keeps a page out of Google? noindex does. robots.txt controls crawling, not indexing. A URL you disallow can still appear in search results if Google finds a link to it somewhere, just without a snippet. To reliably keep a page out of the index you have to let Google crawl it and serve a noindex rule, either as a meta robots tag or an X-Robots-Tag header.

That sentence contains the trap that catches most sites. The instinct, when you want a page gone, is to block it in robots.txt and add a noindex tag for good measure. Belt and braces. Except the belt cuts the braces: if Googlebot is not allowed to fetch the page, it never reads the HTML, so it never sees the noindex you carefully added. You have taken your one working instruction and hidden it behind a locked door.

What is the difference between noindex and disallow?

Disallow is a crawling rule and noindex is an indexing rule. Disallow in robots.txt tells a crawler not to request the URL at all. noindex tells a crawler that has already fetched the page not to include it in search results. One governs the fetch, the other governs what happens after the fetch. They operate at different stages, which is exactly why they are not interchangeable and why combining them backfires.

Think of robots.txt as a sign at the entrance and noindex as an instruction printed inside the building. If the sign turns the crawler away at the door, nobody ever reads the instruction inside.

Does robots.txt prevent indexing?

No. This is stated plainly in Google's own robots.txt specification: Google "can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet." The content stays out. The URL itself can still show up.

The mechanism is straightforward. Crawling supplies the content, but links supply the URL. If three sites link to /secret-pricing/ with the anchor text "secret pricing", Google knows the URL exists and roughly what it covers without ever fetching it, and it can index that URL on the strength of those links alone. What it cannot do is describe the page, so you get a bare listing with no description, which is usually worse than either outcome you wanted.

robots.txt disallow vs meta noindex vs X-Robots-Tag

Three tools, three jobs. Most indexing mistakes come from reaching for the wrong one.

Can I use noindex and disallow at the same time?

You can, and it is self-defeating. The disallow stops Googlebot from fetching the page, so the noindex tag inside it is never read. You end up with the weakest version of both: the page is not crawled, so Google cannot see your removal instruction, and the URL remains eligible for indexing off the back of inbound links. Pick one, and for removal it is always noindex.

The sequence that trips people up looks like this. A page gets indexed. You add Disallow to robots.txt. The page stays in the index, because blocking a crawl does not delete anything already there. You then add noindex, but the disallow is still in place, so Google never returns to read it. The page sits in search results indefinitely while you assume both fixes are working. The way out is to remove the disallow, leave the noindex, and let Google crawl the page one more time so it can act on the instruction.

Can I just put noindex in robots.txt?

No. An unofficial noindex: directive in robots.txt worked for years and Google dropped support for it in September 2019. Any line like that in your file today does nothing. Google supports exactly four fields in robots.txt: user-agent, allow, disallow, and sitemap. Everything else is ignored silently, which is the worst way to be ignored, because your file still parses and you get no warning.

Why does Search Console say "Indexed, though blocked by robots.txt"?

Because you asked Google not to crawl a URL, and it indexed the URL anyway from links pointing at it. That status is Search Console describing the exact scenario above: Google has the address but not the content. It is a warning that your block is doing something other than what you intended.

The fix depends on what you actually wanted. If you want the page out of search, remove the robots.txt block and serve noindex. If you never cared whether the URL appeared and only wanted to stop wasting crawl requests, the status is cosmetic and you can leave it. Deciding which one applies is most of the work, and our guide on why a page is not indexed walks through the neighboring cases.

How do I noindex a PDF or an image?

Use the X-Robots-Tag HTTP header, because a PDF has no <head> to put a meta tag in. Configure your server to send X-Robots-Tag: noindex on the response, and Google drops the file once it recrawls. The same applies to images, videos, and any other non-HTML resource you want kept out of results.

This is also the efficient route for HTML at scale. A single server rule can noindex every URL under /search/ or every file in a directory without touching a template. Just remember the same condition holds: the resource has to stay crawlable, or the header goes unread.

How long does noindex take to remove a page from Google?

It takes as long as the next crawl, which Google says can be months for a page it considers unimportant. noindex is not a delete button you press. It is an instruction that only takes effect the next time Googlebot fetches the URL and reads it. Low-traffic pages get revisited rarely, so the wait can be long.

You can speed it up. Request a recrawl through URL Inspection in Search Console for the specific page. Keep the page linked internally rather than orphaning it, since a page nothing links to is a page Google has little reason to revisit. If the content is sensitive and needs to disappear now, use the Removals tool for a temporary hide and put a real noindex or authentication in place behind it. The removal tool buys you time, it does not do the job.

Which one should I use?

A short decision list covering the cases that come up most:

• Staging or dev sites: use HTTP authentication, not robots.txt. A disallowed staging site still gets its URLs indexed the moment someone links to it, and staging URLs in search results are a genuinely embarrassing way to find this out. Auth removes the possibility entirely.
• PDFs and other non-HTML files: X-Robots-Tag: noindex. There is no other clean option.
• Faceted and filtered URLs: usually robots.txt disallow. These often run into the millions, you do not want them crawled at all, and you rarely care if a stray URL appears. Crawl budget is the real problem here, and disallow is the tool that solves it. Canonical tags are the alternative if you want the crawl to happen but the duplicates consolidated.
• Thin pages, tag archives, internal search results: noindex, and let them stay crawlable. You want Google to see the rule and drop them. Blocking them instead leaves whatever is already indexed stuck in place.
• Private files: passwords, not robots. robots.txt is a public file that lists exactly which paths you consider worth hiding, which makes it a map for anyone curious. It is an honor system that well-built crawlers, from Googlebot to the extraction tools that turn pages into structured data, choose to follow. Anything that ignores it walks straight in.

What robots.txt is genuinely good at

None of this makes robots.txt useless. It is the right tool for crawl control, and on a large site crawl control matters. Point it at the sections that burn requests and return nothing: infinite calendars, session-ID URLs, API endpoints, faceted navigation. Every request Googlebot does not waste there is a request available for pages you want in the index, which is the whole game on a site with more URLs than crawl budget.

A few mechanics worth knowing, straight from Google's spec. Google honors the first 500 kibibytes (KiB) of your robots.txt and ignores anything past that, so keep it tight. It generally caches the file for up to 24 hours, which you can adjust with Cache-Control, meaning a change you push today may not take effect immediately. Two wildcards are supported: * matches zero or more characters, and $ anchors to the end of the URL. Trailing wildcards do nothing, so /fish* and /fish are the same rule. And sitemap belongs in there too, since it is one of the four fields Google reads.

The short version

robots.txt decides what gets crawled. noindex decides what gets indexed. If you block the crawl, you block the only instruction that can remove the page.

When a page has to leave the index, allow the crawl and serve noindex. When a page is fine but not worth the crawl requests, disallow it and accept that a bare URL might surface. When the content is private, neither file is the answer and authentication is.

Indexing keeps track of which of your URLs are actually in Google's index and tells you why the ones that are not are missing, whether that is a robots.txt block you forgot, a noindex tag that Google cannot reach, or a quality call on the content. It monitors coverage continuously, checks page indexing status across your site, and submits eligible URLs through official channels only. No spam, no PBNs, and no pretense that anyone can force Google's hand. Just an accurate picture of what is indexed, and a clear reason for what is not.