Verifying Googlebot and other crawlers: how to tell who is really calling

Typing Googlebot into a user agent costs ten characters, which is why your logs mix the crawler that indexes you with scrapers wearing its name. Verifying Googlebot has nothing to do with reading that header: it means checking where the request actually comes from.

The classic failure has two faces and both are expensive. A firewall may be shutting out the real Googlebot because it looked suspicious, while a scraper walks in freely because it copied the name. One method fixes both, and it is worth applying it before touching any blocking rule.

Why the user agent proves nothing

The User-Agent header is free text chosen by the client. Writing Googlebot/2.1 needs no permission or certificate: it gets copied from a tutorial and pasted into the request. That is why the user agent is only a hint about which address list to consult, never proof of identity.

It helps to separate two questions most tutorials blend together. Verifying answers who is calling; deciding answers whether you want them in. The first is technical and lives in DNS and IP data; the second is a business call, because it affects visibility in search engines and assistants. A genuine Googlebot can still crawl too much, and that gets fixed by adjusting the rate, not by doubting who it is.

The three families of Google requests

Google does not have one crawler, it has three families, and each answers robots.txt differently. Common crawlers, where Googlebot lives, always respect the rules for automatic crawling and resolve as crawl-***-***-***-***.googlebot.com or the geographic variant geo-crawl-***.geo.googlebot.com, with ranges in common-crawlers.json.

Special-case crawlers, such as AdsBot, may or may not respect robots.txt and resolve as rate-limited-proxy-***.google.com. The third family is user-triggered fetchers: a person asked for that action, so they ignore robots.txt and appear as ***.gae.googleusercontent.com or google-proxy-***.google.com. The order matters: if you plan to block, work out the family first.

The reverse DNS mask

The reverse lookup of the IP must end in googlebot.com, google.com or googleusercontent.com. A PTR pointing anywhere else is not Google, no matter how clean the user agent sitting next to it.

Manual verification: how to verify Googlebot in three steps

First find the IP in your access log and run a reverse lookup with host. Second check that the returned name belongs to one of those three Google domains. Third run a forward lookup on that name and confirm it returns the same IP you started with.

The third step is the one almost everyone skips and the one that holds the check together. Verification is a round trip: if you only look at the PTR, an attacker creates their own reverse record with whatever text they like and passes the filter. Google publishes three examples worth keeping: 66.249.66.1 resolves as crawl-66-249-66-1.googlebot.com and returns to the same IP; 35.247.243.240 as geo-crawl-35-247-243-240.geo.googlebot.com; and 66.249.90.77 as rate-limited-proxy-66-249-90-77.google.com. Full detail is in the official verification documentation.

Automated verification: the IP ranges and their new home

Google publishes JSON lists with prefixes in CIDR format, one per family plus goog.json for the rest of its infrastructure. Those lists are the source of truth; the user agent only decides which one to consult. In 2026 they moved: since February 11 the files live under /crawling/ipranges/, and the old path still responds but the documentation recommends updating links.

The common-crawlers.json file currently carries a creation stamp of 10 November 2026 and 317 prefixes across IPv4 and IPv6: it is not a list you copy once and forget, it gets regenerated. Store it in your WAF or compare it inside your log analysis pipeline, and note the file date so you know which version you compared against.

The new case: AI agents and Web Bot Auth

In March 2026 Google added the Google-Agent user agent, used by its agents hosted on Google infrastructure to browse and act on a user's behalf. It brings separate ranges in user-triggered-agents.json. In May it published the Web Bot Auth guide, an experimental protocol built on HTTP message signatures: instead of trusting the name and the IP, the agent signs the request.

You recognise it by the Signature-Agent header carrying g="https://agent.bot.goog", with the signature travelling in Signature and Signature-Input per RFC 9421, and public keys published in the .well-known directory on that same domain. Three limits the documentation states itself: it is experimental, not every request from a given agent is signed, and user agent plus IP verification remains the de facto standard. If your WAF already supports it, switch it on; if not, change nothing this week.

The bots that are not Google: what each vendor publishes

The pattern repeats: a declared user agent plus a vendor IP list. The best documented case is OpenAI, with four agents controlled separately. OAI-SearchBot feeds ChatGPT search; block it and the site stops appearing in those answers, though it may still surface as a navigational link. GPTBot collects content that may be used to train models. ChatGPT-User runs actions a person starts, so robots.txt may not apply. And OAI-AdsBot validates ad landing pages without training on that data.

Two details are worth gold. All four IP files respond today, and gptbot.json lists 18 prefixes stamped September 2026. On top of that, those ranges are not OpenAI's own space: several blocks are registered to Microsoft, which you can confirm through RDAP. That is why blocking "the Microsoft cloud" wholesale takes out OpenAI's crawler along with half the internet. The vendor also warns that a robots.txt change can take about 24 hours to reflect.

How to decide who gets in

With identity settled, three business questions remain. Do I want a search engine to crawl me? Almost always yes. Do I want an assistant to cite me? That is a visibility decision, and discovery traffic looks a lot like search traffic. Do I want my content training models? That is an editorial call, not an SEO one, and it deserves an explicit answer instead of a reflex block.

Googlebot always allowed: if there is a spike, throttle the rate. Assistant search bots allowed if you want citations, as covered in the GEO guide. Training bots are your call, and worth knowing: Google-Extended has no request user agent of its own, it is only a robots.txt token and does not affect inclusion or ranking in Google Search. User-triggered fetchers cannot be blocked with robots.txt; if you must slow them, it happens server side.

What is coming in 2027

Google announced it plans to support the shared, machine-readable AIPREF signal with the __usage_ai-training token in early 2027. It is an announcement, not something available today.

The four mistakes that actually cost traffic

The first is blocking by user agent while believing you are verifying: the scraper keeps coming and you keep the feeling of having done something. The second is blocking by ASN or cloud provider, because Google and OpenAI ranges share space with third-party services and you discard the good with the bad. The third is shutting Googlebot out at the WAF during a spike, precisely when new pages matter most.

The fourth is the most expensive and the most common. Google states it plainly: do not use 401 and 403 to limit crawl rate, because 4xx codes other than 429 have no effect on crawl rate and, worse, crawlers report to the next processing system that the content does not exist. A defensive 403 does not slow anything down: it can push pages out of the index.

If you really must slow things down: 429 or 503 with Retry-After

For an emergency lasting hours or a couple of days, Google documents returning 500, 503 or 429 instead of 200 and attaching Retry-After with seconds or an absolute UTC date. Honestly: sustained 5xx or 429 keeps indexed URLs around for a while and eventually removes them. It is for an emergency, not a way of life. The code details live in the 5xx and 429 guide.

How to check it on your site this week

Three cheap checks. In the log, count requests whose user agent says Googlebot and work out what share resolves to googlebot.com on reverse DNS: anything that does not resolve is imitation, and it is worth knowing how much. Then download common-crawlers.json and compare it with the IPs you see, keeping the file date. Finally review crawl stats in Search Console and use the URL Inspection API after any WAF or CDN change.

The closing idea is about process, not tooling: write a policy stating who gets in, who does not and which status code slows things down, then review it twice a year and every time a vendor changes its list. None of this raises rankings. It stops you losing them by accident, which is a different thing and also counts.