Noindex Detected in X-Robots-Tag HTTP Header

This message means your server is returning the response header X-Robots-Tag: noindex for that URL, which tells search engines to keep it out of results. The rule is in the HTTP response, not in the HTML, so it will not appear in view-source. In most cases nobody on the team added it deliberately: a CMS setting, an SEO plugin, a leftover server config block or a CDN rule is emitting it.

You will meet the message in one of two places. Lighthouse and PageSpeed Insights report it under the indexing audit as "'noindex' detected in 'X-Robots-Tag' http header". Google Search Console reports the same URL as excluded by a noindex tag. They are describing one condition, and both are right even when you are certain your code contains nothing of the kind.

Why you cannot find it in your page

Because it is not in your page. A robots rule can travel two ways: as a meta tag in the HTML head, or as an HTTP response header. The X-Robots-Tag HTTP header is the second route, and it carries exactly the same rules as the meta tag. Google puts it plainly in its robots meta tag documentation: any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.

The practical difference is visibility. A meta tag sits in the document, so it shows up in view-source, in your template, in a diff and in almost any audit tool. A header sits in the response, above the document, and none of those places will show it to you. Search engines read both with equal weight. Humans only ever see one.

That asymmetry is the entire reason this bug survives for months. A team checks the HTML, finds nothing, concludes the problem must be something else, and starts investigating crawl budget or backlinks while a single response header quietly keeps a section of the site out of Google.

Confirm the header is really there

Start by verifying the live response rather than trusting a report that may be hours or days old. Open your browser devtools, go to the network tab, reload the page, click the document request at the top of the list, and read the response headers. If the rule is present you will see a line reading X-Robots-Tag: noindex, often as noindex, nofollow.

From a terminal, a HEAD request does the same job in one line and is easier to repeat across several URLs:

curl -sI https://example.com/your-page | grep -i x-robots-tag

Two details are worth getting right at this stage. First, check more than one URL. Header rules are applied by pattern, so they usually hit an entire directory or file type rather than the single page you happened to test. Checking the one URL in the report tells you nothing about how wide the damage is.

Second, request the origin as well as the public URL. If a CDN sits in front of your site, the header you see from your laptop may have been added at the edge and may not exist on your server at all. Requesting the origin directly, bypassing the CDN, splits the problem in half immediately: if the header is present at the origin the cause is in your application or server config, and if it is absent there but present publicly, the cause is at the edge.

Where the header actually comes from

Work outward from the application toward the network. In rough order of how often each one turns out to be responsible:

• A CMS visibility setting. WordPress has an option under Settings, Reading called "Discourage search engines from indexing this site". It is meant for sites under construction, it is easy to leave on after launch, and it emits the header without touching a template.
• An SEO plugin rule. Plugins let you noindex by post type, taxonomy, author archive or template. A rule scoped to one of those will affect a whole section and look like nothing at all in the page source.
• Theme or application code. A single line such as header("X-Robots-Tag: noindex, nofollow") in a functions file or a middleware layer, usually added for a legitimate reason years ago and never scoped back down.
• Server config. An Apache Files block or an nginx location block written for a staging environment that survived promotion to production. This is the most common cause on self-hosted sites.
• A CDN or edge rule. A Cloudflare transform rule, a CloudFront response headers policy or an edge function adding the header after your origin has already responded correctly.
• A platform default. Managed hosts, app platforms and some ecommerce platforms apply the header to preview URLs, staging subdomains and certain system paths by default. Nobody on your team did anything, and the header is still there.

Fix it in the right order

The sequence matters more than it looks, because two of these steps are commonly done too early and waste days.

1. Identify the layer. Compare the origin response with the public response, as above. Do not start editing until you know which layer is adding the header.
2. Remove or scope the rule. Turn off the CMS setting, correct the plugin rule, or narrow the config pattern. If the rule was there for a reason, scope it rather than deleting it, so the staging environment keeps the protection it needs.
3. Purge every cache in front of it. This is the step people skip. A cached response carrying the header will keep being served after the origin is fixed, so you will retest, see no change, and conclude the fix did not work. Purge the CDN, the page cache and any reverse proxy.
4. Verify the live response. Run the HEAD request again against the public URL and confirm the header is gone. Check a handful of URLs across the affected pattern, not just the one from the report.
5. Only then ask Google to recrawl. Requesting indexing while the header is still being served accomplishes nothing except spending one of a small daily allowance of requests. Fix first, verify, then submit.

What happens after you remove it

Nothing instant. Google has to fetch the URL again before it can see that the rule is gone, so recovery depends on recrawl timing rather than on your deploy. Pages Google visits often can return within days. URLs it rarely crawls can take weeks, and a page that was dropped from the index has to be rediscovered and reprocessed rather than simply switched back on.

You can shorten that considerably. Make sure the affected URLs are in your XML sitemap, link to them from pages Google already crawls frequently, and submit them through the official channels rather than waiting. A URL indexing tool that resubmits through the Google Indexing API and Bing IndexNow, then watches for the status to change, turns an open-ended wait into something you can actually track.

Watch for one specific trap while you wait. If the affected URLs are also disallowed in robots.txt, removing the header changes nothing, because Google cannot fetch the page to discover that anything changed. Crawl permission has to come first. Google says the same thing about the rules themselves: these settings can be read and followed only if crawlers are allowed to access the pages that include them.

When the header is correct and the report is not a problem

Not every instance of this message is a bug. Lighthouse flags the condition, not the intent, so a page that is supposed to be excluded will still be reported. Checkout steps, account pages, internal search results, thank-you pages, print views and filtered listing URLs all have good reasons to carry noindex, and seeing the audit fail on them is the tool working correctly.

The same is true of files. A PDF has no HTML head, so the header is the only way to keep a published document out of search results at all. That is exactly the case Google names when it says you can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags in HTML is not possible. Contracts, pricing sheets and internal collateral published to a public path belong in that category, and the documents you deliberately keep out of Google still need to be findable by the people who work with them, which is a job for search across your own internal tools and drives rather than for a public search engine.

So the question to ask is not whether the header is present but whether it is present on the right URLs. Read the pattern rather than the page. A rule matching /documents/ when it should have matched /documents/internal/ looks identical in the report and costs you an entire section.

Stop it from coming back

This failure repeats because the changes that cause it are rarely filed as SEO work. A plugin update, a config promotion, a CDN rule added during an incident, a platform default that shifts after a migration: none of those arrive with a ticket saying they might deindex the site.

Two habits close most of the gap. Add a production check for the header to your deploy process, so a staging rule that travels with config gets caught at release rather than at the next traffic review. And monitor the directives your live URLs serve on a schedule, alongside whether Google and Bing actually hold each URL, because reading the two together is what separates a page that is correctly excluded from a page that is missing by accident. A standing noindex checker across the whole URL set turns a quarter of lost traffic into an alert within days, and it catches the header version of the rule that source-code audits never see.

If you want the background on the directive itself, including the exact configuration for Apache, nginx, IIS, Cloudflare and the common frameworks, and the rules on combining directives or scoping them to a single crawler, that is all on the X-Robots-Tag reference. And if Search Console is the place you met this, our write-up on excluded by noindex tag covers how the same condition is reported there.