# Log File Analysis for SEO: See What Crawlers Actually Do

**Author:** John Morabito (Founder, /winston)
**Published:** September 19, 2026
**Reading time:** 10 minutes
**Canonical:** https://www.winstondigitalmarketing.com/playbooks/log-file-analysis-for-seo/

Almost every technical SEO diagnosis runs on secondhand evidence. Search Console tells you what Google decided to report back, aggregated and delayed. A crawl tool tells you what a crawler would find if it behaved the way your crawl tool behaves, which makes it a simulation of your own settings. Both are useful. Neither is a record of what happened.

Your server logs do. Every time anything requests a URL on your site, the server writes a line about it: when, which URL, which user agent asked, which IP it came from, and what came back. Filter that record down to the bots and you are looking at the only unmediated account of what search engines and AI systems really did on your site. Most people never open it, which is why it is still a good place to find a problem nobody else has spotted.

## What is actually in a log line

A standard access log line is boring. It carries a timestamp, the requesting IP, the method, the requested path including any query string, the status code returned, the response size, usually a referrer, and the user agent string. One line per request, and a busy site produces millions of them, so the work is less about reading logs than grouping them. The analysis is always the same shape: filter to the crawlers you care about, group requests by URL pattern or template, then compare that distribution against the one you wanted. The gap is the finding.

## What logs tell you that nothing else will

Start with frequency, per URL. You can see that a category page is fetched several times a week while a product page two clicks away has been fetched once in the last month. Search Console will not give you that per-URL history, and no simulated crawl can, because frequency is a behavior rather than a property of the page.

Then there are the URLs you did not know you had. Logs routinely surface patterns that surprise the people who own the site: parameter combinations from a filter nobody uses, old URLs from a platform replaced years ago, a calendar generating a page for every month into the far future. You cannot find these by crawling from the homepage, because nothing links to them. They show up in the log because a bot found them once and kept coming back.

Logs also give you the status codes bots actually received. What you see in a browser is not always what a bot gets. Logs show the real response: the redirect chain that resolves in three hops, the 5xx window during a deploy nobody noticed, the page returning 200 with an empty error template.

## Where crawl attention gets wasted

The most common finding on a large site is that a big share of bot requests goes to URLs that should never be crawled. The usual suspects are predictable once you know to look:

- Filter and sort combinations from faceted navigation, which multiply into an enormous URL space from a modest catalog.
- Tracking and session parameters appended to otherwise fine URLs, creating a distinct URL per visitor path.
- Internal site search result pages, generated on demand and effectively infinite.
- Paginated archives running far deeper than anyone intended.
- Legacy URLs that still resolve through a redirect, so the bot spends a request to be told where to go.

Seeing it in a log is different from suspecting it. You get the actual proportion of requests that went to each pattern, and that number turns the fix into an easy decision instead of an argument. The faceted case has its own remedies, which I wrote up in [faceted navigation SEO](https://www.winstondigitalmarketing.com/playbooks/faceted-navigation-seo/). The log tells you that you have the problem; that playbook tells you what to do about it.

## The pages that get crawled rarely, or never

The mirror image of waste is neglect. Pull the list of URLs you consider commercially important, then check how many times each was requested in your window. Every time I have done this on a large site, some page that matters has been fetched once, or not at all.

A page crawled rarely will be slow to reflect any change you make and slow to recover if it slipped. The cause is usually structural: it sits too deep, or too few internal links point at it, or the links that do live on pages that are themselves rarely crawled. The log gives you the symptom and your internal link graph gives you the cause. The same wiring governs whether AI crawlers can reach and make sense of a page, which I covered in [internal linking for AI crawlers](https://www.winstondigitalmarketing.com/playbooks/internal-linking-for-ai-crawlers/).

## Verify the bot before you believe it

This is the step people skip, and skipping it poisons everything downstream. The user agent string is text the requester chose to send, and anyone can send one that says Googlebot. Some share of the traffic in your log claiming to be a major search crawler is not that crawler.

There are two accepted ways to check. The first is a forward-confirmed reverse DNS lookup: resolve the requesting IP back to a hostname, resolve that hostname forward again, and confirm you land on the same IP and that the hostname sits on the operator's documented domain. The second is matching the IP against the ranges the operator publishes. Google and Microsoft both document verification for their crawlers, and several AI crawler operators publish IP ranges too.

> If you have never run this check, run it first. Unverified bot traffic is worth knowing about on its own, since it consumes server resources you pay for. But the reason to separate it out early is that every conclusion about crawl allocation is wrong if a third of your Googlebot lines were not Googlebot.

## The AI crawlers are in the same file

Server logs are the most direct evidence available about whether AI systems are fetching your content, because the retrieval side of those systems is otherwise opaque. You can see which AI user agents requested which pages and what status they got back. If one is being served an error, or is blocked by a robots.txt rule somebody added years ago for a different reason, the log is where you find out.

Be careful what you conclude from it. A fetch is a fetch. It tells you your content was retrievable and that something retrieved it. It does not tell you the content was used, cited, or shown to anyone, and anybody selling you a straight line from crawl to citation is filling in a gap the data does not cover. Treat the log as evidence of access, which is a precondition for citation rather than proof of it.

## When this is worth doing, and how to get the files

Not always, and I would rather say so than sell the exercise. Log analysis earns its cost when crawl allocation is a real constraint or when something specific has gone wrong: large sites with many thousands of URLs, ecommerce catalogs with faceted navigation, publishers with deep archives, any CMS capable of generating URLs without limit. It also earns it in the weeks after a migration, when you want to know whether bots found the new structure and what they hit along the way. If you run a forty-page site with no known crawl problem, skip it. Your constraints are elsewhere.

When you do want the files, ask whoever runs the infrastructure for access logs covering a defined window, a few weeks on a busy site and a couple of months on a quieter one. Two cautions. Know where to ask: if a CDN sits in front of your site, cached requests never reach the origin, so origin logs alone understate crawl activity and you want the edge logs too. And treat the files as sensitive, because they contain IP addresses and query strings that can carry personal data. Handle them under the retention and privacy rules you already follow, and think before uploading raw logs to a third-party tool.

## Turning the file into work

A log analysis that ends in a chart is a wasted afternoon. The output should be a short list of changes, and three cross-references get you most of the way:

- Crawled but not in your sitemap. URLs the engine knows about and you do not manage. Decide whether each pattern should exist, be consolidated, or be blocked.
- In your sitemap but rarely crawled. Pages you claim matter that the site is not supporting. Fix the depth and the internal linking.
- Crawled often but earning nothing. Compare crawl share by template against impressions by template. A template absorbing heavy crawl while producing no visibility is either the wrong thing to have indexed or a sign of duplication.

Each of those produces concrete tasks: consolidate a URL pattern, fix a redirect chain, add internal links to a neglected page, stop generating a family of URLs nobody needs. If you want a starting point before going this deep, the [90-minute technical SEO audit](https://www.winstondigitalmarketing.com/playbooks/technical-seo-audit-90-minutes-claude/) covers the checks worth running first. Log analysis is the layer you add when those checks say something is wrong and you need to know exactly what.

We do this as part of our [SEO service](https://www.winstondigitalmarketing.com/services/seo/), usually on large catalogs and after migrations, where the gap between what a site thinks is happening and what is actually happening tends to be widest.

## Frequently asked questions

### What is log file analysis in SEO?

Log file analysis means reading your server's access logs to see exactly which URLs crawlers requested, when, how often, and what status code they received. Every request to your site writes a line in that log, including requests from search engine bots and AI crawlers. Analysis means filtering those lines down to the bots you care about, grouping the requests by URL pattern or template, and comparing that picture to what you expected. It answers questions no other tool can answer directly: which of your pages get crawled often, which get crawled almost never, what junk URLs are eating crawl attention, and what errors bots hit that a human browsing the site would never see.

### What can server logs show you that Search Console cannot?

Search Console reports crawl activity in aggregate and with a delay, and it reports what Google chooses to expose rather than the raw record. It will not give you a per-URL request history, and it says nothing at all about other crawlers. A crawl tool has the opposite problem: it shows you what a simulated crawler finds, which reflects your configuration rather than real behavior. Logs are the actual record. They show you request-by-request detail for every bot that reached your server, including URL patterns you did not know existed, the exact status codes returned, and the frequency with which any single page is fetched. Use Search Console for what Google reports about indexing, and use logs when you need to know what really happened.

### How do you verify that a crawler is really Googlebot?

You verify by IP address, not by user agent, because the user agent string is simply text the requester chose to send and anyone can put Googlebot in it. There are two accepted methods. The first is a forward-confirmed reverse DNS lookup: resolve the requesting IP back to a hostname, then resolve that hostname forward again and confirm it returns the same IP, and check that the hostname belongs to the crawler operator's documented domain. The second is matching the IP against the list of crawler IP ranges the operator publishes. Google and Microsoft both document verification for their crawlers, and several AI crawler operators publish IP ranges as well. Do this before drawing conclusions, because a meaningful share of self-declared bot traffic is somebody else's scraper wearing a familiar name.

### Which sites actually need log file analysis?

Sites where crawl allocation is a real constraint or where something has gone wrong. That means large sites with many thousands of URLs, ecommerce catalogs with faceted navigation and parameter sprawl, publishers with deep archives, and any site where the CMS can generate URLs without limit. It also means any site immediately after a migration, a redesign, or a big structural change, because logs are the fastest way to see whether bots found the new structure and what they hit on the way. A small brochure site with fifty pages and no known crawl problem does not need this. Reach for logs when you have a specific question about crawler behavior, not as routine maintenance.

### How do you get server log files, and what should you be careful about?

Ask whoever runs your hosting or infrastructure for access logs covering a defined window, usually a few weeks to a couple of months depending on traffic. Where you request them matters: if a CDN sits in front of your site, requests served from cache never reach the origin server, so origin logs alone will understate crawl activity and you want the CDN's logs too. Be careful on two fronts. Logs contain IP addresses and can contain query strings with personal data, so treat them as sensitive, handle them under your organization's retention and privacy rules, and do not paste raw logs into tools without knowing where the data goes. And keep the window bounded, because a year of logs on a busy site is enormous and rarely tells you more than a representative recent sample.