Check & monitor · X-Robots-Tag
X-Robots-Tag Noindex: Set the HTTP Header, Check What Your URLs Serve, and Fix Pages Google Will Not Index
Every technical SEO has met this bug. The page looks perfect. The HTML has no noindex, the robots.txt allows it, the sitemap lists it, and Google still refuses to index the URL. The rule is there, it is just not in the document. It is in the response header, one line above everything you were looking at.
Submit · monitor coverage · official methods only
Ready to check coverage
Paste a sitemap to sweep every URL for index status, then submit the missing ones through the official Google Indexing API and Bing IndexNow.
Coverage
indexed
Avg time to index
URLs submitted
Now eligible
Live, interactive · sample data · official methods only
Official Google Indexing API · Bing IndexNow · verified sitemaps · no spam, no PBNs
In short
The X-Robots-Tag is an HTTP response header that carries the same indexing rules as the robots meta tag, but in the server response instead of the HTML. Google documents it plainly: "The X-Robots-Tag can be used as an element of the HTTP header response for a given URL," and "Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag." The usual syntax is X-Robots-Tag: noindex. Its real advantage is reach: you can apply it to files that have no HTML head at all, which is why Google says "You can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags in HTML is not possible." The catch is that it is invisible in view-source, so a header set by a CDN, a hosting default or a stray plugin can keep a page out of Google for months without leaving a single visible trace on the page.
Last updated August 2026
That is the whole problem with the X-Robots-Tag: it is the most powerful indexing control you have and the easiest one to lose track of, because nothing in the browser shows it to you. A hosting provider ships it as a default on a staging subdomain. A CDN rule applies it to a path pattern nobody documented. A theme file adds it in PHP. Six months later a section of the site has quietly vanished from search and the audit that checked the HTML found nothing wrong.
This page is the reference and the check in one place. Below you can read exactly what the header does, the config that sets it correctly in Apache, nginx, IIS, Cloudflare and the common frameworks, how it differs from robots.txt and the meta tag, and what to do about the Lighthouse and Search Console message telling you a noindex was detected in an X-Robots-Tag http header. Then run your URLs through the console above: Indexing reads the indexing directives each URL actually serves, header and meta together, confirms whether Google and Bing really have the page, names the reason for every gap, and resubmits the pages that should be live through the official Google Indexing API, Bing IndexNow and your sitemaps.
Official methods only
White hat · no spam, no PBNs
Why it works
What your team gets with x-robots-tag
The rule view-source cannot show you
A noindex in the HTML is one keystroke away from being spotted. The same rule in a response header is invisible in the page source and survives every audit that only reads markup.
The only option for files without a head
A PDF, an image or a CSV export has nowhere to put a meta tag. The header is the one place a rule can go, which is why it governs so much of what a real site actually publishes.
What you set versus what Google got
Reading your own config tells you what you intended. Only checking the live response and the live index status tells you what your CDN, host and cache actually served.
What it handles
Submitted, monitored and fixed, automatically
Indexing submits your URLs through the official Google Indexing API, Bing IndexNow and clean XML sitemaps, watches coverage across both engines, and flags any page that drops out with a plain-English reason so you can resubmit and get it back.
- Reads the X-Robots-Tag header and the robots meta tag each URL actually serves, together
- Catches a noindex applied by a CDN, host default or cache layer that never appears in your code
- Confirms whether Google and Bing still hold the URL, so you know if the rule ever landed
- Covers PDFs, images and other non-HTML files that meta tag audits skip entirely
- Flags the header plus robots.txt disallow conflict that leaves URLs stuck in the index
- Resubmits the pages you meant to keep through the official Indexing API, IndexNow and sitemaps
Not indexed yet
/blog/seo-guide-2026 is discovered but not indexed
thin content signal, queued for re-crawl via the Indexing API
Why Indexing
One place to submit, monitor and fix coverage
Not a black-hat indexer that risks your site, not a free checker that only tells you the bad news. Indexing unifies official submission and live coverage monitoring, the white-hat way, across Google and Bing.
Submits the official way
Bulk-submit through the Google Indexing API, Bing IndexNow and clean XML sitemaps. We speed discovery and re-crawl using methods the engines support, never spam, PBNs or black-hat tricks.
Monitors coverage live
You do not refresh a search bar one URL at a time. Indexing watches which pages are in Google and Bing, catches anything that drops out, and tracks time-to-index across your whole site.
Diagnoses and resubmits
Every non-indexed page comes with a plain-English reason, then auto-resubmits through the official API so it gets another shot. Google still decides, but nothing waits in the dark.
At a glance
How to set the X-Robots-Tag on every common stack
The header name and values are identical everywhere; only the config syntax changes. The Apache and nginx examples below are the ones Google publishes in its own robots meta tag documentation. Apply the rule as narrowly as you can: a pattern that is one character too broad is how whole sections disappear.
| Stack | Where the rule goes | Example that noindexes PDFs |
|---|---|---|
| Apache | .htaccess or httpd.conf, inside a Files or FilesMatch block | <Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </Files> |
| nginx | The server or location block in your site config | location ~* \.pdf$ { add_header X-Robots-Tag "noindex, nofollow"; } |
| IIS | web.config, under system.webServer httpProtocol customHeaders | Add a custom header named X-Robots-Tag with the value noindex, nofollow |
| Cloudflare | Transform Rules, Modify Response Header, matched on a URI path expression | Set static header X-Robots-Tag to noindex when the path ends in .pdf |
| Node and Express | Middleware, before the static file handler runs | res.set("X-Robots-Tag", "noindex, nofollow") |
| Next.js | The headers function in next.config.js, with a source pattern | Return a header X-Robots-Tag with value noindex for the matching source |
| WordPress | A functions.php hook, or your SEO plugin, which may already be setting it | header("X-Robots-Tag: noindex, nofollow") |
| CloudFront and CDNs | A response headers policy or edge function on the distribution | Add X-Robots-Tag as a custom response header on the behavior serving the files |
What the X-Robots-Tag header is, and when you need it instead of a meta tag
The X-Robots-Tag is a line in the HTTP response your server sends before the page content. Google describes it as an element of the HTTP header response for a given URL, and the syntax is as plain as it looks: X-Robots-Tag: noindex, sitting among the Content-Type and Cache-Control headers a browser never displays.
It is not a different system from the robots meta tag. It is the same rule set delivered by a different transport. Google states it directly: any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag. So noindex, nofollow, none, nosnippet, noarchive, notranslate, noimageindex, max-snippet, max-image-preview, max-video-preview, indexifembedded and unavailable_after all work in the header exactly as they work in the head of a document.
The reason to reach for the header is coverage. A meta tag needs an HTML head to live in, and a large share of what a real site serves has no head at all. Google names the case explicitly: you can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags in HTML is not possible. Price lists, whitepapers, invoices, CSV exports, generated images, sitemaps you would rather not have indexed as pages, all of them are reachable only through the header.
The second reason is scope. A meta tag has to be added to each document, usually by a template. A header rule is a pattern match in one config file, so it applies to a whole directory, file type or hostname at once. That is a strength when you are retiring a section and a liability when the pattern is looser than you thought.
- Same rules as the meta tag, delivered in the HTTP response instead of the HTML
- The only way to set an indexing rule on a PDF, image or other non-HTML file
- Applies by pattern, so one line can cover an entire path or file type
- Invisible in view-source, which is exactly why it goes unnoticed for so long
- Set per file type, per directory or per hostname depending on your server
X-Robots-Tag vs robots.txt vs the robots meta tag: which one to reach for
These three get treated as interchangeable ways to keep a page out of Google, and they are not. Two of them control indexing. One controls crawling. Confusing the last with the first two is the single most common way a page nobody wanted in search stays in search anyway.
robots.txt controls whether a crawler is allowed to fetch a URL. It says nothing about whether that URL may appear in results. If Google is blocked from fetching a page but finds enough signals pointing at it, the URL can still be listed. Worse, blocking it guarantees the opposite of what people intend, because Google can never read the noindex you put on the page. Google is explicit about the dependency: these settings can be read and followed only if crawlers are allowed to access the pages that include these settings.
That gives you a clear order of operations when a URL is both blocked and carrying a noindex. Unblock it in robots.txt first, let Google fetch it and read the rule, wait for the URL to drop out of results, and only then re-block it if you still want to save the crawl. Doing those steps in the wrong order leaves the page sitting in the index indefinitely, which is the state most people arrive in.
Between the meta tag and the header, the choice is mostly practical. Use the meta tag when the page is HTML and the decision belongs to that one document, because it lives with the content and moves with it. Use the header when the file has no head, when the rule covers a pattern rather than a page, or when you do not control the template but do control the server or the CDN.
- robots.txt: controls crawling only, never a reliable way to remove a URL from results
- Robots meta tag: controls indexing, HTML documents only, lives with the content
- X-Robots-Tag: controls indexing, works on any file type, applies by pattern
- A blocked page can never have its noindex read, so the two cancel each other out
- Unblock, let Google read the noindex, confirm the drop, then re-block if you want to
Fixing noindex detected in X-Robots-Tag http header when you never set it
This message reaches people from two directions: Lighthouse and PageSpeed Insights report it under the indexing audit, and Search Console reports the URL as excluded by a noindex tag. Both are telling you the same thing, and both are correct even when you are certain your code does not contain it. The header is almost never where you are looking, because whatever is adding it is usually not in your repository.
Work outward from the file rather than inward from the symptom. Start with the application: a CMS setting such as the WordPress option to discourage search engines, or an SEO plugin rule scoped to one post type, will emit the header without touching a template. Then the server config, where a Files or location block written for a staging environment often survives a promotion to production. Then the edge, where a CDN transform rule or a response headers policy adds the header long after your origin has finished responding.
Two things make this harder than it should be. The first is caching: if a cached response carrying the header is still being served, you can fix the origin and see no change at all until the cache is purged. The second is that platform defaults change without you doing anything, which is why the header appears on Shopify, managed WordPress hosts and app platforms in cases where nobody on the team edited a line.
The order that resolves it fastest is boringly mechanical. Confirm the header is really on the live response rather than a stale report. Identify which layer adds it by requesting the URL directly from the origin, bypassing the CDN. Remove it at that layer. Purge every cache in front of it. Verify the header is gone from the live response. Only then ask Google to recrawl, because requesting indexing while the header is still being served just burns the request.
- Check the CMS first: a discourage search engines setting or a plugin rule needs no template edit
- Then server config, where a staging Files or location block often survives promotion
- Then the CDN, where a transform rule adds the header after the origin has responded
- Purge caches before retesting or you will keep reading the old response
- Verify the header is actually gone before requesting a recrawl, not after
How to check what X-Robots-Tag your URLs are actually serving
Checking one URL is easy and checking a site is not, which is the gap that lets this bug live for months. A single request shows you the response headers immediately: open the network panel in your browser devtools, click the document request, and read the response headers, or run a HEAD request from the command line and read what comes back. Either tells you the truth about that one URL at that one moment.
The trouble starts at scale, and scale is where the header actually bites, because it applies by pattern. A rule that matches one path prefix hits every URL underneath it, so the pages that go missing are a whole section rather than a page you would happen to spot check. Sampling ten URLs from a site of forty thousand is how a section stays gone until someone notices the revenue.
There is also a difference between what your server sends and what Googlebot received. A CDN can vary responses by geography, by user agent or by cache state, so the header you see from your laptop is not automatically the header Google got. That is why the useful check pairs the directive with the outcome: read what the URL serves, then confirm whether the URL is actually in the index. A page serving no noindex but missing from Google is a different problem from a page serving a noindex and correctly absent, and the fix is different too.
Run that pairing on a schedule rather than during incidents. Header rules get introduced by config changes, plugin updates and CDN edits that nobody files as SEO work, so the failure mode is silent by design. Monitoring the whole URL set turns a quarter of lost traffic into an alert within days.
- Read response headers in devtools or with a HEAD request for a single URL
- Check the pattern, not a sample, because header rules apply to whole paths at once
- Request from the origin as well as through the CDN, since the edge can differ
- Pair the directive you read with real Google and Bing index status
- Monitor continuously, because the change that breaks it is rarely filed as SEO work
Targeting a single crawler, and combining several rules in one header
The header supports more precision than most people use. Google notes that the X-Robots-Tag may optionally specify a user agent before the rules, which lets you write a rule that applies to one crawler and leaves the others alone. The value is written with the crawler name first, then a colon, then the rules, as in googlebot: noindex. Any crawler not named keeps the unqualified behavior.
This is genuinely useful when your requirements differ per engine. You may want a document out of Google Search but still reachable by an internal or partner crawler, or you may want a rule to apply to Googlebot News and not to Googlebot. Be careful with the assumption that other engines read the qualifier the same way. Bing honors the X-Robots-Tag, but the safe assumption for any crawler outside Google is that it reads the unqualified rules and may ignore a name it does not recognize.
Combining rules is equally straightforward. Google states that multiple X-Robots-Tag headers can be combined within the HTTP response, or you can specify a comma-separated list of rules. So X-Robots-Tag: noindex, nofollow and two separate header lines carrying noindex and nofollow mean the same thing. When rules end up in conflict, Google resolves it in the direction of caution: the more restrictive rule applies. A page carrying both index and noindex will be treated as noindex.
One rule deserves a specific mention because it is the only one with an expiry built in. The unavailable_after rule tells Google not to show the page in results after a given moment, with the date and time in a widely adopted format such as RFC 822, RFC 850 or ISO 8601. It suits content with a genuine end date, such as an event page or a time-limited offer, and it saves you from remembering to add a noindex later.
- Name a crawler before the rules to scope a directive, as in googlebot: noindex
- Unnamed crawlers fall back to the unqualified rules in the same header
- Combine rules with commas, or send several X-Robots-Tag headers, both are valid
- Where rules conflict, the more restrictive one wins
- unavailable_after sets an expiry date using RFC 822, RFC 850 or ISO 8601 format
Using the header deliberately: the cases where it is the right tool
Most writing about the X-Robots-Tag treats it as a hazard to be found and removed. It is also a tool, and there are situations where nothing else does the job. Knowing them is what keeps you from reaching for robots.txt and getting the result nobody wanted.
Gated and internal documents are the clearest case. Sales collateral, pricing sheets, contracts and internal PDFs get published to a public path so a link can be shared, then get discovered by Google because a URL only has to be mentioned once. There is no head to put a meta tag in, and blocking the path in robots.txt does not remove anything already indexed. A header rule on the file type or directory is the only control that both applies to the file and gets read.
Generated and duplicate output is the second. Filtered listing URLs, print views, export endpoints, image derivatives and paginated archives multiply fast and add nothing to search. Setting noindex by pattern removes them from results while leaving them fetchable, which matters because Google needs to fetch the URL to see the rule at all.
Staging and preview environments are the third, and the one that most often goes wrong in the other direction. A blanket header on a staging hostname is exactly right. The failure is not the rule, it is the promotion: the config moves to production with the rule attached, and the live site inherits a noindex nobody meant to ship. If you use this pattern, and you should, make the presence of the header on production an explicit check in your deploy process rather than something you trust yourself to remember.
- PDFs and other non-HTML files you published but do not want in search results
- Filtered, print, export and image derivative URLs that add nothing to the index
- Time-limited pages, using unavailable_after instead of a manual cleanup later
- Staging and preview hostnames, scoped by host so production cannot inherit it
- Make a production header check part of deploy, since this rule travels with config
Keep reading
- Noindex Detected in X-Robots-Tag HTTP Header Your server is sending X-Robots-Tag: noindex, so the rule never appears in view-source. Here is where it comes from and the order that actually fixes it.
- Excluded by Noindex Tag: Meaning and How to Fix It (2026) Google crawled the page, found a noindex rule and left it out on purpose. The eight real causes, why the header version is invisible in view-source, and the fix for each.
- Robots.txt vs Noindex: Which Keeps a Page Out of Google? Robots.txt controls crawling, noindex controls indexing. Blocking a page hides your noindex tag from Google. Here is which one actually removes a page, and when.
Good questions
Questions about x-robots-tag
Explore more
More ways teams get every page indexed
301 redirect SEO
Point Google at the new URL, then confirm it actually moved there.
Learn moreNofollow and dofollow links
The attribute everyone reaches for to control ranking, which actually changes what gets crawled.
Learn moreInternal linking for SEO
The cheapest indexing lever most sites never pull, and the reason half their pages were never crawled.
Learn moreStop guessing. Get every page indexed and keep it that way.
Bulk-submit your URLs through the official Google and Bing channels, monitor coverage, and resubmit anything that drops out, automatically. White hat only, so we speed discovery without ever guaranteeing what Google chooses to index.
Google Indexing API · Bing IndexNow · sitemaps · coverage monitoring · official methods only