Check & monitor · Googlebot
Googlebot: User Agent Strings, IP Ranges, and How the Google Crawler Reads Your Site
Most indexing problems are really crawling problems wearing a disguise. A page sits at Discovered, currently not indexed for six weeks and the team rewrites the copy, when the actual cause was a firewall answering Googlebot with a 403 while every human browser got a 200. Server logs hold the answer, and almost nobody reads them, partly because half the traffic claiming to be Googlebot is not Googlebot at all: the user agent string is a plain text header and anyone can copy it.
Submit · monitor coverage · official methods only
Ready to check coverage
Paste a sitemap to sweep every URL for index status, then submit the missing ones through the official Google Indexing API and Bing IndexNow.
Coverage
indexed
Avg time to index
URLs submitted
Now eligible
Live, interactive · sample data · official methods only
Official Google Indexing API · Bing IndexNow · verified sitemaps · no spam, no PBNs
In short
Googlebot is the web crawler Google uses to discover and fetch pages for Search. It runs in two variants, Googlebot Smartphone and Googlebot Desktop, both identified by the user agent token Googlebot, and it crawls almost every site from the mobile variant first. Google publishes the IP ranges its crawlers use in common-crawlers.json, and the only reliable way to confirm a visitor really is Googlebot is a reverse DNS lookup that resolves to googlebot.com, google.com or googleusercontent.com plus a matching forward lookup. Crawling is not indexing: Googlebot fetching a page only makes it eligible, and Google still decides separately whether the page enters the index.
Last updated July 2026
This page is the reference for what Googlebot actually is: the user agent tokens, the full strings, where Google publishes its IP ranges, how to verify a crawler properly, and what Googlebot does after it fetches your HTML. Indexing then handles the part logs cannot tell you. It checks whether the pages Googlebot crawled ever made it into the Google index, names the ones that did not with a plain-English reason, resubmits the genuine gaps through the official Google Indexing API, Bing IndexNow and your sitemaps, and tracks how long each one takes to land.
Official methods only
White hat · no spam, no PBNs
Why it works
What your team gets with googlebot
Know which crawler actually visited
Googlebot, GoogleOther and Google-InspectionTool do different jobs and mean different things in your logs. Reading them as one bot hides the signal.
Verify before you block or whitelist
The user agent header is trivially spoofed. Reverse plus forward DNS, or Google's published IP ranges, is the only proof that holds.
Crawled is not indexed
Logs prove Googlebot fetched the page. Only an index check proves Google kept it, and that is the gap this tool closes.
What it handles
Submitted, monitored and fixed, automatically
Indexing submits your URLs through the official Google Indexing API, Bing IndexNow and clean XML sitemaps, watches coverage across both engines, and flags any page that drops out with a plain-English reason so you can resubmit and get it back.
- Confirms whether pages Googlebot crawled actually entered the Google index
- Names the reason each crawled page was skipped, in plain English
- Resubmits genuine gaps through the official Indexing API, IndexNow and sitemaps
- Monitors thousands of URLs so a crawl regression shows up in days, not quarters
- Tracks time to index so you can prove a crawl fix worked
Not indexed yet
/blog/seo-guide-2026 is discovered but not indexed
thin content signal, queued for re-crawl via the Indexing API
Why Indexing
One place to submit, monitor and fix coverage
Not a black-hat indexer that risks your site, not a free checker that only tells you the bad news. Indexing unifies official submission and live coverage monitoring, the white-hat way, across Google and Bing.
Submits the official way
Bulk-submit through the Google Indexing API, Bing IndexNow and clean XML sitemaps. We speed discovery and re-crawl using methods the engines support, never spam, PBNs or black-hat tricks.
Monitors coverage live
You do not refresh a search bar one URL at a time. Indexing watches which pages are in Google and Bing, catches anything that drops out, and tracks time-to-index across your whole site.
Diagnoses and resubmits
Every non-indexed page comes with a plain-English reason, then auto-resubmits through the official API so it gets another shot. Google still decides, but nothing waits in the dark.
At a glance
Google crawler user agents, what each one does, and what it means in your logs
The user agent token is the string you match in robots.txt. The full user agent string is what appears in server logs. Several Google crawlers also obey the Googlebot token, which is why a broad robots.txt rule can affect more than you intended.
| Crawler | robots.txt token | What it does | Seeing it in logs means |
|---|---|---|---|
| Googlebot Smartphone | Googlebot | The primary crawler for Search. Fetches pages as a mobile device and is the version used for indexing on virtually every site. | Your normal Search crawl. This is the traffic that matters most for indexing. |
| Googlebot Desktop | Googlebot | The desktop variant of the same crawler. Still used, but a minority of fetches since mobile-first indexing completed. | Routine crawling. Nothing unusual on its own. |
| Googlebot Image | Googlebot-Image | Crawls images for Google Images. Also obeys rules written for the Googlebot token. | Image discovery. Blocking it removes you from image search, not web search. |
| Googlebot Video | Googlebot-Video | Crawls video files and video pages. Also obeys the Googlebot token. | Video indexing activity. |
| Googlebot News | Googlebot-News | Crawls for Google News. Uses the Googlebot user agent string and also obeys the Googlebot token. | News surface crawling, relevant to publishers. |
| Google-InspectionTool | Google-InspectionTool | Fetches pages for the URL Inspection tool and the Rich Results Test. Also obeys the Googlebot token. | Somebody clicked Test Live URL. Not a scheduled crawl. |
| GoogleOther | GoogleOther | A generic crawler used by internal Google teams for research and development fetches, separate from Search. | Not Search crawling. It does not affect indexing, and blocking it will not hurt rankings. |
| Google-Extended | Google-Extended | A robots.txt control token only, with no separate user agent string. Governs use of your content for Gemini model training and grounding. | Nothing appears in logs. It is a policy switch, not a crawler. |
| Storebot-Google | Storebot-Google | Crawls product and checkout pages for Google shopping surfaces. | Ecommerce product data collection. |
What is Googlebot?
Googlebot is the name for Google's web crawling software: the program that follows links, fetches URLs, and hands the results to the systems that decide what goes into the Search index. It is not one machine but a large distributed fleet running from thousands of IP addresses, which is why the crawler that hit your site this morning and the one that hit it this afternoon almost certainly came from different addresses.
There are two variants that matter for Search: Googlebot Smartphone and Googlebot Desktop. Both use the user agent token Googlebot in robots.txt, so you cannot write a rule that applies to one and not the other. Since mobile-first indexing finished rolling out, the smartphone variant is what Google indexes with on effectively every site, which has a practical consequence people still get wrong: if content, links or structured data are missing from the mobile rendering of a page, they are missing from indexing, no matter how complete the desktop version looks.
Googlebot also renders. It fetches the HTML, queues the page, then runs a recent version of Chrome against it to execute JavaScript and see the final DOM. Rendering is not instant and not guaranteed to be identical to your browser, so JavaScript-injected content, client-side routing and lazy-loaded links are the usual suspects when a page crawls fine but indexes with the wrong content or does not index at all.
- Two Search variants: Googlebot Smartphone (primary) and Googlebot Desktop.
- Both share the robots.txt token Googlebot. You cannot separate them by rule.
- Mobile rendering is the version that gets indexed, on virtually every site.
- Googlebot renders JavaScript with a recent Chrome, but on a separate, slower queue.
- Googlebot crawls over HTTP/1.1 and HTTP/2, and generally accepts gzip and Brotli compression.
The Googlebot user agent strings, and why matching on them is not enough
The desktop crawler sends a user agent along the lines of Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36, where the Chrome version rolls forward over time. The smartphone crawler sends a Nexus 5X Android string with the same Googlebot/2.1 fragment appended. Because the Chrome version changes, any code that matches on the full string will eventually break. Match on the substring Googlebot instead, and treat that match as a hint rather than proof.
It is only a hint because a user agent is just a request header. Anyone can send it. Scrapers routinely announce themselves as Googlebot precisely because so many sites whitelist that string, skip rate limits for it, or serve it the full unpaywalled page. If your analytics show a surge of Googlebot traffic that your Search Console crawl stats do not confirm, you are almost certainly looking at impostors, and any bandwidth or performance decision based on that number will be wrong.
The reverse case is more damaging and much more common: a bot management layer, a WAF rule or an aggressive rate limiter decides the real Googlebot looks suspicious and starts serving it 403s, CAPTCHAs or a JavaScript challenge. Everything looks fine to you because your browser gets a 200. In Search Console the pages drift to Crawled, currently not indexed or simply fall out of the index, and nothing in the interface names the security layer as the cause.
- Match the substring Googlebot, never the full user agent string, which changes.
- A user agent header is trivially spoofed and proves nothing on its own.
- Fake Googlebot traffic inflates bandwidth stats and fools whitelists.
- A WAF or bot manager blocking the real Googlebot is a top cause of silent deindexing.
- Test with the URL Inspection live test, which fetches as Google-InspectionTool.
How to verify Googlebot: IP ranges and reverse DNS
Google publishes two ways to confirm a crawler is genuine, and both are authoritative. The first is a reverse DNS check, which Google describes as the manual method. Take the IP address from your logs and run a reverse lookup on it. The hostname must end in googlebot.com, google.com or googleusercontent.com, in patterns like crawl-66-249-66-1.googlebot.com or rate-limited-proxy-66-249-66-1.google.com. Then run a forward lookup on that hostname and confirm it resolves back to the exact IP you started with. If either step fails, the request was not from Google, whatever the user agent claimed.
The second method is matching against Google's published IP ranges. Google maintains JSON files listing the address blocks its crawlers use, and the one covering Googlebot is common-crawlers.json under developers.google.com/static/crawling/ipranges/. Related files cover special crawlers, user-triggered fetchers such as Google Site Verifier, and the fetchers that act on behalf of users. These files change, so treat them as a feed to refresh on a schedule rather than a list to copy once into a config file and forget.
For anything larger than a spot check, automate it. Push the verification into your log pipeline so every request claiming to be Googlebot is tagged verified or spoofed at ingest, and you get a real number for Google crawl volume instead of a guess. Once you trust that number, Search Console's Crawl Stats report becomes genuinely useful: your logs show what was requested, Crawl Stats shows how Google saw the responses, and a gap between them usually points at something sitting between your server and the crawler.
- Reverse DNS must resolve to googlebot.com, google.com or googleusercontent.com.
- Always follow with a forward lookup that returns the original IP. One direction is not enough.
- Google publishes crawler IP ranges as JSON, including common-crawlers.json for Googlebot.
- Refresh the IP lists on a schedule. They are not static.
- Never whitelist on user agent alone, and never rate limit a verified Googlebot.
How often Googlebot crawls, and what actually changes it
There is no fixed schedule and no setting to request a faster one. Google decides crawl rate from two things: how much your server can take, and how much Google wants your content. The capacity side is automatic and reactive. Fast, stable responses let the crawl rate rise. Slow responses, timeouts and a run of 5xx errors make Googlebot back off within hours, which is why a bad deploy can cost you crawl volume for days after the site is healthy again.
The demand side is about worth. Popular URLs, pages that change in ways Google can verify, and sites that publish consistently get crawled more. Stale content, duplicated pages, endless parameter and faceted URL variants and thin auto-generated pages drag it down, because Google learns that fetching your URLs mostly returns nothing new. This is also why the old Crawl Rate Limiter tool was removed from Search Console in January 2024: Google's position is that Googlebot already adapts to your server automatically, so the manual throttle was doing more harm than good.
What you actually control is narrow but real. Keep server response times low and error rates near zero. Stop wasting fetches on URLs that should never be crawled, using robots.txt for infinite parameter spaces rather than noindex, which still costs a fetch. Return accurate 404 or 410 status codes for pages that are gone instead of soft 404s. Keep sitemaps clean and lastmod honest. Then measure: if you cannot see time to index moving, you are guessing about whether any of it helped.
- No crawl schedule exists, and no button makes Googlebot come sooner.
- Persistent 5xx responses or slow replies cut crawl rate automatically, within hours.
- The Search Console Crawl Rate Limiter tool was removed in January 2024.
- robots.txt saves crawl budget. noindex does not, because the page still has to be fetched.
- Crawl budget only becomes a real constraint on large or fast-changing sites.
From crawl to index: where pages fall out
Googlebot fetching a URL is step one of several, and every later step can drop the page. After the fetch, Google renders it, canonicalizes it against near-duplicates, evaluates whether it is worth storing, and only then indexes it. A page can be crawled cleanly every week and never appear in Search because Google picked a different URL as canonical, or judged the content too thin or too similar to be worth a slot.
That is why log analysis alone gives an incomplete picture, and why the two most common Search Console statuses look so alike but need opposite fixes. Discovered, currently not indexed means Googlebot knows the URL exists but has not spent a fetch on it yet, which is a crawl demand and internal linking problem. Crawled, currently not indexed means the fetch happened and Google chose not to keep the page, which is a content, duplication or canonical problem. Resubmitting the URL is useless for the second one.
Indexing sits on the other side of that gap. It takes your URL list or sitemaps, checks real index status for every URL in Google and Bing, names the reason for each miss, and resubmits the pages that genuinely deserve another look through the official Google Indexing API, Bing IndexNow and your sitemaps, then watches until the status changes. What it will never do is promise forced indexing, because no tool can deliver that. Google decides. The value is knowing exactly which pages are missing and why, on day two rather than next quarter.
- Crawled does not mean indexed. Rendering, canonicalization and quality checks come after.
- Discovered, currently not indexed is a crawl demand problem. Fix internal links and value.
- Crawled, currently not indexed is a content or duplication problem. Resubmitting will not help.
- Only an index check confirms the outcome. Logs stop at the fetch.
- Watch the trend across thousands of URLs, not one page at a time.
Good questions
Questions about googlebot
Explore more
More ways teams get every page indexed
Stop guessing. Get every page indexed and keep it that way.
Bulk-submit your URLs through the official Google and Bing channels, monitor coverage, and resubmit anything that drops out, automatically. White hat only, so we speed discovery without ever guaranteeing what Google chooses to index.
Google Indexing API · Bing IndexNow · sitemaps · coverage monitoring · official methods only