Server Log Analysis for SEO: What Google Crawls

Search Console tells you how many pages Google crawled, but not every request: it aggregates, samples, and only covers your own domain. Your server logs keep the exact hour, URL, and status code. Server log analysis for SEO is the difference between guessing and knowing.

What server logs are and why they are the crawl source of truth

An access log is a list of lines: one per HTTP request your server receives. In the combined format, each line carries the origin IP, date and time, the method, the requested path, the status code, the response size, and the declared user-agent. It is the raw record of what happened, with no interpretation and no sampling. That is its advantage over any dashboard: tools aggregate and estimate, while the log keeps the individual event with its exact time. When you need to know whether Google crawled a specific page again after a change, the answer is in the log, not in a seven-day average.

Where logs live and how rotation works

On Apache they live in access.log; on Nginx, in the file of the same name inside the server log directory. On managed hosting you download them from the panel or over FTP. The detail that breaks analyses is rotation: logs get split and compressed into .gz files, so a two-week window may be spread across several files. Merge them and sort by date before concluding anything, or you are looking at a fragment of the picture.

Logs vs. Search Console: sample vs. reality

The Crawl Stats report has three limits worth knowing: it only exists for root properties, it counts every redirect hop as a separate request (301, 301, and 200 are three rows for one logical visit), and its data is sampled. Google adds that a site with fewer than a thousand pages should not need that report. Translated: logs are not for browsing for the sake of it, but for diagnosing a specific problem. When something does not add up, the log is the next level of evidence.

How to verify a request really comes from Google (and not a fake bot)

Google warns in its crawler verification guide that the user-agent string can be spoofed. Anyone can write Googlebot in their HTTP client and show up in your logs looking like the search engine. If you decide based on that label, you work with data polluted by scrapers and third-party tools.

Manual method: reverse DNS plus forward DNS

The official verification has two steps. First you look up the reverse DNS of the IP and check that the name ends in googlebot.com, google.com, or googleusercontent.com. Then you look up the forward DNS of that name and confirm it returns the same IP you started with. If both steps do not close, the request is not from Google. The examples the search engine publishes: IP 66.249.66.1 resolves to crawl-66-249-66-1.googlebot.com, 35.247.243.240 to geo-crawl-35-247-243-240.geo.googlebot.com, and 66.249.90.77 to rate-limited-proxy-66-249-90-77.google.com.

Automated method: the IP range JSON files

For large volumes, Google publishes five files with the ranges in CIDR format: common crawlers, special crawlers, user-triggered fetchers, Google fetchers, and agents. The catch is that they are updated almost daily, so a list copied from a blog goes stale within weeks. In a recent check the common crawlers file held 317 prefixes and the agents file had grown from 4 to 20 since it was created in March 2026. The DNS mask for common crawlers follows the pattern crawl-something.googlebot.com or geo-crawl-something.geo.googlebot.com. If you compare thousands of IPs against DNS, cache the result per IP: repeating the lookup for the same address is wasted time.

The Google crawlers you will see in your logs (and what each one means)

Not everything carrying Google's signature is search crawling, and mixing up the categories leads to wrong conclusions about your indexation. The official list groups visitors into three families worth separating from the first filter.

Search crawling: the one that affects your SEO

Googlebot, in its smartphone and desktop versions, feeds Search, Discover, Images, Video, and News. Its Googlebot-Image and Googlebot-Video variants handle those formats, and Storebot-Google crawls for Shopping. One case that surprises people: Googlebot-News has no user-agent of its own and crawls with the regular Googlebot strings, so you will not tell a news visit apart in the log. These are the requests that do influence what your users see.

Internal tools: not search crawling

Google-InspectionTool is the agent used by URL Inspection and the rich results test. GoogleOther and its variants are generic crawls run by product teams for internal research. Seeing them in your logs does not mean Google is deciding about your indexation: it means someone inside Google is consulting your page. Treating them as Googlebot distorts any crawl frequency report.

Fetchers and AI agents

This is the new development of the year. Google-Agent, documented since March 2026, corresponds to agents that browse the web and take actions at a person's request. The family also includes Google-GeminiNotebook (formerly Google-NotebookLM), Feedfetcher, Google-CWS, GoogleMessages, and Google-CloudVertexBot.

The detail that confuses people most: user-triggered fetchers ignore robots.txt. It is not a configuration error: the request comes from a person who wants to view or summarize your page, not from a crawler deciding to index you. Blocking them bluntly when the goal was to slow down training crawlers is aiming at the wrong target.

Server log analysis for SEO step by step: from raw file to decisions

Step 1: prepare the data

Merge the rotated files for the period and normalize the format into something queryable (CSV or parquet). Filter out static assets when your goal is to analyze HTML. And separate verified bots from those that merely declare themselves as such: match the IPs against the range JSON or your DNS cache. That filter turns a list of lines into a reliable sample.

Step 2: the metrics that matter

With clean data: requests per day and per crawler; status code distribution (200, 301, 302, 404, 5xx); response time by status code; most crawled URLs; crawl frequency of the pages you care about; and first and last visit to new or updated pages, which answers how long Google takes to come back after a change.

Step 3: cross-reference with Search Console

Logs and Search Console answer different questions. Crawl Stats provides trend and availability; Indexing says what was left out and why; Performance shows which crawled pages earn clicks. The log explains the why behind a variation; the tool supplies the how much.

Step 4: five patterns that expose problems

Heavy crawling of parameter URLs or of facets with no unique content: the crawl budget is spent on variants that will never rank.
Spikes of 5xx errors or timeouts during peak traffic: Google slows its crawl rate and your updates take longer to show up.
404 errors on URLs that have internal links: these are the ones Search Console stops reporting because the crawler grows tired of them.
Redirect chains visible as 302, 302, and 200: they burn requests and dilute part of the signal.
Key pages crawled very little or never while the crawler wanders through irrelevant pages: the cause is usually internal linking or a robots.txt directive.

Common log-reading mistakes that produce false conclusions

Trusting the user-agent label without verifying DNS or IP.
Lumping GoogleOther, Google-InspectionTool, and Googlebot together. Only the last one is search crawling.
Blocking AI fetchers when the goal was to slow down training crawlers.
Drawing conclusions from a single day: an algorithm update, a traffic spike, or a deployment explains one-off variations.
Forgetting that IP addresses are personal data: anonymize them, limit retention, and never publish a raw log.

When to automate (and when Search Console is enough)

If your site has fewer than a thousand pages, start with Crawl Stats and URL Inspection. Automate when recurring indexing problems appear, when you handle a huge number of URLs, or when you need alerts, for example a warning if crawling of an entire section drops. Approaches go from light to heavy: a small script that turns logs into CSV or parquet; a dump into a database or an analytics warehouse to build dashboards; and log analysis suites when the volume justifies them.

Checklist for your first analysis

Download the log window you want to study.
Merge and normalize the files for the period.
Verify the IPs against DNS or the range JSON files.
Split the requests by crawler.
Calculate status codes and the most crawled URLs.
Cross-reference your findings with Search Console.
Define one concrete action for each problem.

For background, review what technical SEO covers and how crawl budget is spent. If the log shows 404s or redirect chains, there are guides for 404 and soft 404 errors and for the 301 redirect, and it is worth contrasting conclusions with this Search Console guide. Before you trust a crawl block, review your robots.txt: the logs will tell you whether that rule does what you thought.