# Which AI Crawlers Actually Visit Your Site (and What They Take)

**Author:** John Morabito (Founder, /winston)
**Published:** September 20, 2026
**Reading time:** 11 minutes
**Canonical:** https://www.winstondigitalmarketing.com/playbooks/ai-crawler-log-analysis/

Almost every argument about GEO is conducted on inference. Whether the models can see you, whether your content is being taken, whether something you blocked last year has been costing you ever since. There is one place where the answer is a recorded fact rather than a theory, and it is the server log nobody opens.

## The short answer

AI crawler log analysis is three steps. Filter your server access logs down to the agents the AI companies publish. Verify that those requests actually came from the operators they claim to come from. Then split what is left into two piles: bulk collection, and live retrieval by an agent fetching one page because somebody asked a question a second ago.

That second pile is the one attached to your business. Everything else in GEO is inference. A visibility tracker samples what an engine said to prompts you picked, which is useful and is still a sample of a moving target. Analytics sees only the people who clicked through, which is a fraction of the people who got an answer. Your logs are different in kind. They are a first-party record of what your server was asked for, by whom, and what it gave back.

The general mechanics of working with logs, getting the files, grouping requests by pattern, finding crawl waste and bad status codes, are covered in log file analysis for SEO: https://www.winstondigitalmarketing.com/playbooks/log-file-analysis-for-seo/ . This page is the AI layer on top of it.

## The agents, and what each one is for

The major operators do not run one crawler each. They run several, doing different jobs, and the difference between them is most of the value in this exercise.

OpenAI publishes GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic publishes ClaudeBot, Claude-SearchBot and Claude-User. Perplexity publishes PerplexityBot and Perplexity-User. The pattern across all three is the same: one agent gathering content in bulk, one building a search index, and one fetching a page on behalf of a person who is mid-conversation. Apple, Meta, Amazon and ByteDance each run their own crawlers, and Common Crawl's CCBot collects content that ends up in datasets a great many other people train on, which is why blocking one company's crawler while leaving CCBot open is such a common inconsistency.

Google is the one that confuses everybody. Google-Extended is not a crawler. It is a token you can address in robots.txt to tell Google whether content that Googlebot already fetched may be used for Gemini. Nothing ever requests a page as Google-Extended, so it will never appear in your log. The practical consequence is a real blind spot: Googlebot's request line does not tell you which Google product consumed the page afterwards, so the Google side of your AI exposure is not visible in logs the way the others are.

Two habits keep this list from going stale. Build your filter from each operator's current documentation rather than a list somebody pasted into a spreadsheet a year ago, because agents get added and renamed. And keep an explicit bucket for bot traffic you have not classified, so the next new agent shows up as an unknown you can go look at instead of disappearing into the general noise.

## The user agent string is a claim, not a fact

A user agent is text the requester chose to send. Anyone can send a request that says ClaudeBot or GPTBot, and it takes one command-line flag to do it. Some share of what is in your log claiming to be an AI crawler is not one. Sometimes that is a research project or a monitoring tool, and sometimes it is a scraper that picked a name it expects you to allow.

There are two accepted ways to check. The first is a forward-confirmed reverse DNS lookup: take the requesting IP, resolve it back to a hostname, resolve that hostname forward again, confirm you land on the same IP, and confirm the hostname sits on the operator's documented domain. The second is matching the IP against the ranges the operator publishes, which most of the AI companies now provide as a downloadable list for each of their agents. Some operators support both, and where they do, the IP list is the faster thing to automate.

> Verify, then count. A report that says an AI crawler visited four thousand times is not a finding if some of those lines were somebody else's scraper. Unverified bot traffic is worth knowing about on its own, since it uses server capacity you are paying for, but the reason to separate it early is that every conclusion about AI crawl activity is built on top of it.

## Two patterns: the sweep and the fetch

A collection crawler sweeps. Requests spread broadly across your URL space, spaced out over hours or days, following whatever order its queue is in rather than anything happening in the world. It works through breadth. It is indifferent to whether you published something this morning.

A retrieval agent arrives as a burst of one. A single URL, or two or three, requested within seconds of each other, then nothing for days. There is no crawl path leading to it, because nothing crawled its way there. The agent went straight to the address because a model produced that address while a person sat waiting for an answer.

If you have never watched this happen, it is worth doing once. Tail a live log, open an assistant, ask it something about your own company, and watch the request land. It takes two minutes, and it is the most convincing demonstration of GEO I know of, because it is the one part of the whole discipline you can watch happen in front of you.

The split is not perfectly clean. Search-index crawlers sit between the two: they gather in bulk, but what they gather is what gets cited later. A retrieval agent may pull several URLs for one question. So sort by the job the operator says each agent does, and use the request shape as corroboration rather than the other way round.

## Separating them in a report

The reason to keep the two piles apart is arithmetic. Bulk crawling generates far more requests than retrieval does, so anything that averages them together buries the only signal tied to a human asking a question. Build the report as two tables.

The collection table answers one question: are the agents reaching the pages you want represented, and are they getting real responses? Requests by agent, how much of your URL space each one reached, status codes, and the templates taking the most attention.

The retrieval table answers a different one: which of your pages are being pulled into live answers, and is that changing? Individual fetches by URL and timestamp, grouped by agent, tracked over time. A third column worth adding is whether each fetched page is one you were actually trying to make citable, because the mismatch between those two lists is usually the most interesting thing in the report.

One more cut belongs in both: status codes grouped by verified agent.

## When a bot you want is getting a 403

This is the most common finding on the AI side of a log review, and almost nobody chose it deliberately. The block is nearly always somewhere that nobody thinks of as a content decision:

- CDN bot management with a rule that blocks or challenges an entire category, including agents you would rather serve.
- A WAF rule matching on something crude, like the word bot in the user agent string.
- Rate limiting that trips when an agent requests several pages in quick succession, which is exactly what a retrieval agent does.
- A robots.txt disallow inherited from an old configuration, written for a reason nobody remembers.
- Geographic or ASN blocking that happens to cover the ranges an operator runs from.
- A consent screen, paywall, or interstitial that returns a 200 with none of the actual content behind it. This is the worst case, because it does not look like a failure anywhere in your reporting.

You find all of these the same way: group status codes by verified agent and look for a wall of 403s, 429s, or suspiciously uniform response sizes. Then fix it at the layer the block actually lives on, which is often not the layer people go to first.

Which brings up the reason edge logs matter more here than in ordinary SEO work. If your CDN refuses a request, it never reaches the origin, so in origin logs an agent you are blocking looks exactly like an agent that never came. Absence and refusal are precisely the two findings you need to tell apart, and origin logs cannot do it. Pull the edge logs. And treat all of these files as sensitive while you have them, because they contain IP addresses and query strings that can carry personal data.

Deciding what to allow is a separate question from discovering what you are currently blocking, and it does not have to be all or nothing. Blocking bulk collection while allowing retrieval is a coherent position, and robots.txt can express that agent by agent. The difference between the files that control this, and the one that does not control anything, is covered in llms.txt vs robots.txt: https://www.winstondigitalmarketing.com/playbooks/llms-txt-vs-robots-txt/ . Whatever you decide, confirm it in the log afterwards rather than trusting the configuration screen, since the point of this exercise is that configuration and behavior are not the same thing.

## From a retrieval fetch to a citation you can check

Here is the honest version of connecting a log line to an outcome. It is a chain of evidence rather than a measurement, and it is still far better than the guessing it replaces.

1. Record the fetch: which agent, which URL, what time, what status code came back.
2. Go to the same engine reasonably soon after and ask the question that fetch plausibly came from. Use the engine that matches the agent, since a Perplexity fetch tells you nothing about what ChatGPT would say.
3. Record whether your page is cited, and capture the answer, because it will not be the same next week.
4. Cross-reference referral traffic. Some assistants pass a referrer when a user clicks a citation, and the setup for catching that in analytics is in how to measure AI search traffic in GA4: https://www.winstondigitalmarketing.com/playbooks/how-to-measure-ai-search-traffic-ga4/ .

The fetch is certain. The link between the fetch and the answer you saw is an inference, and repeating it is what makes it stronger. What you can say honestly at the end is something like: this page is fetched regularly by retrieval agents, and when we run the prompts we track, it is cited for these.

## What the log proves, and what it does not

| Claim | Does the log support it? |
| --- | --- |
| An AI agent requested this URL | Yes, once the IP is verified |
| The agent received the content | Yes, from the status code and response size |
| A person was waiting for it | Probably, for the user-triggered agents, since that is what they are for |
| The content was used in the answer | No |
| The content was cited | No, you have to go look at the answer |
| This produced a visit | No, that is a referrer and analytics question |
| Our content is in a training set | No, collection is not the same as inclusion |
| A block is costing us citations | Not directly, but a verified agent getting 403s is a real cause with a real fix |

A fetch is a fetch. It proves access, and access is a precondition for citation rather than evidence of it. That boundary is what makes the method worth trusting. A page that has never been fetched is certainly not being cited. A page that gets fetched constantly and never cited has a content problem rather than an access problem. Knowing which of those two you are holding is worth the afternoon on its own.

## Where to go from here

The sequence is short. Get the logs, including the edge logs if a CDN is in front of you. Filter to the AI agents using each operator's current documentation. Verify by reverse DNS or published IP ranges before counting a single line. Split collection from retrieval. Group status codes by agent and fix anything you are blocking that you did not mean to block. Then start tracking retrieval fetches for the pages you care about, so next quarter you have a comparison instead of a snapshot.

Run it again after any change to the CDN, the WAF, the bot rules, or robots.txt, because those are the changes that cut you off without anyone noticing, and they are made by people who are not thinking about AI visibility at the time.

If you run this and find no verified AI agent traffic at all, that is a finding rather than an empty result. Either something is blocking them or nothing has found you yet, and the first of those is usually fixable this week.

We do this as part of our GEO service: https://www.winstondigitalmarketing.com/services/generative-engine-optimization/ .

## Frequently asked questions

### What is AI crawler log analysis?

AI crawler log analysis is reading your server access logs to see which AI agents requested which URLs, when, and what status code came back. The work has three steps. Filter the log down to the agents the AI companies publish, verify that the requests really came from those operators rather than from something wearing their name, then split what remains into bulk collection and live retrieval. It is the only first-party evidence available in GEO. Visibility trackers sample what an engine says to prompts you chose, and analytics only sees the people who clicked through. The log records what your own server was asked for.

### Which AI crawlers should I look for in my server logs?

Each of the major operators runs several agents with different jobs, so look for the set rather than one name. OpenAI publishes GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic publishes ClaudeBot, Claude-SearchBot and Claude-User. Perplexity publishes PerplexityBot and Perplexity-User. Apple, Meta, Amazon and ByteDance each run their own, and Common Crawl's CCBot gathers content that ends up in datasets many others use. Google is the exception worth knowing about: Google-Extended is a robots.txt token rather than a crawler, so it never appears in a log at all. Build the filter from the operators' current documentation, because names get added and changed, and keep a bucket for unclassified bots so new agents surface instead of disappearing into the noise.

### How do you verify that an AI crawler is real and not a spoofed user agent?

By IP address, never by the user agent string, because that string is just text the requester chose to send and anyone can put ClaudeBot or GPTBot in it. There are two accepted checks. The first is a forward-confirmed reverse DNS lookup: resolve the requesting IP back to a hostname, resolve that hostname forward again, confirm you land on the same IP, and confirm the hostname sits on the operator's documented domain. The second is matching the IP against the ranges the operator publishes, which most of the AI companies now do as a downloadable list. Run this before you count anything. Every conclusion you draw about AI crawl activity is wrong if a meaningful share of the lines were somebody else's scraper wearing a familiar name.

### What is the difference between an AI training crawler and a retrieval agent?

A training or indexing crawler is gathering content in bulk to build something. It sweeps broadly across your URL space, works through its own queue on its own schedule, and pays no attention to what is happening in the world at that moment. A retrieval agent fetches a page because a person asked a question seconds ago and is waiting for the answer. In the log that shows up as a single URL, or a small handful, requested in a burst, with no crawl path leading to it because the agent went straight to the address. The two transactions mean different things to your business. Collection takes your content and gives you nothing back directly. Retrieval is the moment your page has a chance to become a cited source in front of someone actively looking for what you sell.

### Why would my site be blocking AI crawlers when I never blocked them?

Because the block usually lives somewhere nobody thinks of as a content decision. The common sources are a CDN bot management setting that blocks a whole category, a WAF rule matching on the word bot in the user agent, rate limiting that trips when an agent requests several pages in a burst, a robots.txt disallow inherited from an old configuration, or a consent or login interstitial that returns a 200 with none of the actual content in it. That last one is the most dangerous, because it does not look like a failure in any report. The way to find all of them is to group status codes by verified agent. A crawler you want that is collecting 403s or 429s is a decision somebody made without knowing they were making it.

### Can server logs prove that an AI assistant cited my page?

No, and anyone selling you a straight line from fetch to citation is filling in a gap the data does not cover. A log line proves that an agent requested a URL and what your server returned. It does not prove the content was used, that it appeared in an answer, or that a human ever saw it. What you can do is build a chain: note the retrieval fetch with its agent and timestamp, ask the same engine the question it plausibly came from, record whether the page is cited, and cross-reference any referral traffic in analytics. That chain is inference supported by first-party evidence, which is better than the inference alone. Treat the fetch as proof of access, which is a precondition for citation rather than a substitute for checking.

### Do I need CDN logs, or are origin server logs enough?

For AI crawler work you want the edge logs, and it matters more here than in ordinary SEO log analysis. If a CDN sits in front of your site, anything served from cache or refused at the edge never reaches the origin, so an agent your CDN is blocking looks identical in origin logs to an agent that never visited. Absence and refusal are the two findings you most need to tell apart, and origin logs cannot separate them. Pull the edge logs, and treat all of these files as sensitive, since they contain IP addresses and query strings that can carry personal data.

### How often should you run AI crawler log analysis?

Quarterly is enough for most sites, with one exception that matters more than the schedule: run it after any change to the CDN, the WAF, the bot management settings, or robots.txt. Those changes are what silently cut off an agent, and they are made by people who are not thinking about AI visibility when they make them. The first run is the valuable one, because it tells you whether you are reachable at all. After that you are watching two things: whether retrieval fetches for the pages you care about are trending anywhere, and whether anything has started getting blocked since you last looked.