# Your WAF Is Quietly Blocking the AI Crawlers You Invited

**Author:** John Morabito (Founder, /winston)
**Published:** September 20, 2026
**Reading time:** 12 minutes
**Canonical:** https://www.winstondigitalmarketing.com/playbooks/waf-and-cdn-blocking-ai-crawlers/

Somebody wrote a robots.txt that welcomes GPTBot, ClaudeBot and PerplexityBot. Somebody else, earlier, and quite possibly a vendor nobody works with anymore, switched on a bot management default at the CDN. The second one wins every time, because it answers the request before your server hears about it.

## The short answer

robots.txt is a document your origin server publishes. Your CDN and your web application firewall sit in front of that server and answer requests on their own authority. When a bot rule at the edge decides an incoming request looks automated, it returns a challenge or a refusal without ever asking your origin what it thinks, and without ever reading the file where you granted permission. The permission is real. It is just never exercised.

So the crawl-control file everyone argues about is the second layer, and the first layer is a setting in a dashboard that a marketing team usually does not have a login for. If you have spent any time deciding which AI agents to allow, and you have not checked what your edge does to those agents, you have been tuning a radio that is unplugged.

The companion to this page is AI crawler log analysis: https://www.winstondigitalmarketing.com/playbooks/ai-crawler-log-analysis/ , which is about seeing the problem in your own records. This one is about the layer where the problem gets created, and it can be checked in about two minutes without access to any logs at all.

## What this looks like on a real site

On September 20, 2026 we ran our automated on-page audit against a New York dispensary site as part of routine prospect research. The tool is a single-page read: it fetches one URL the way a retrieval agent would, checks what came back for the things that make a page quotable, and scores it out of sixteen. This site scored one.

The score came from a 202 with an empty body. A 403 would have been caught by somebody eventually, because errors get looked at. A 202 sits in the two hundreds, so every uptime monitor reports the site as healthy, and an automated fetch comes back holding nothing.

That site's robots.txt explicitly allowed the AI crawlers. Someone had done the work. There was FAQ schema on the page too, which is the kind of thing you add specifically so that a model can lift a clean answer out of it. None of it was ever read, because the edge answered first and the origin never got the chance.

> What that score does and does not tell you: it is one automated fetch of one URL. It says nothing about that site's traffic, its rankings, or whether any assistant has ever been asked about it. What it measures is whether a machine asking politely for the page gets the page, and the answer was no. We have no idea how long that had been true, and neither did the site.

## Why a CDN default outranks anything you write in robots.txt

It helps to be concrete about the order of operations, because once you have it, the rest of this page is obvious.

1. A crawler sends a request for a URL on your domain.
2. DNS points at your CDN, so the request lands at an edge node, not at your server.
3. The edge scores the request. It looks at the user agent, the IP and its reputation, whether the address belongs to a data center, the TLS fingerprint, the request headers and their order, and whatever history it has with that client.
4. If the score crosses a threshold, the edge responds on its own. Refusal, challenge, rate limit, or an interstitial that returns quickly and contains nothing.
5. Only requests that survive step four are passed to your origin server, which is the only machine in this chain that has ever seen your robots.txt.

robots.txt is also a voluntary protocol in the first place. It works because well-behaved crawlers choose to read it and respect it, and the major AI operators do. That cooperation is worth something, and it is worth exactly nothing when the crawler never gets a response to cooperate with. A well-behaved agent that receives a 403 does the polite thing and backs off, which is to say your edge has trained it to stop asking.

The thing that makes this hard to catch is that nobody in the conversation is wrong. The person who wrote the robots.txt did it correctly. The person who turned on bot protection was defending the site from something real. The two decisions were made in different rooms, months or years apart, and the interaction between them was nobody's job.

## The responses that mean challenge, not page

Here is the set to recognize. Only the first two show up as failures anywhere, which is why the other four tend to last for years.

| What comes back | What it usually means |
| --- | --- |
| 403 Forbidden | A flat refusal, from a WAF rule or a bot category block. The honest version, and the easiest to find. |
| 429 Too Many Requests | Rate limiting tripped by a burst. Retrieval agents fetch a few pages in seconds, which is exactly the shape rate limiters are built to stop. |
| 503 with a script in the body | An interactive challenge. A browser would run the script and get through. A crawler cannot, so the 503 is where it ends. |
| 200 whose body is a challenge widget | A success code carrying a verification page instead of your content. Everything downstream counts this as a working page. |
| 202 with an empty body | Accepted and answered with nothing. The case above. Invisible in every uptime and error report you already read. |
| 200 that is far smaller than the real page | A consent gate, a login wall, or a shell that fills in after JavaScript runs. Related to, though not the same as, the problem in JavaScript SEO: https://www.winstondigitalmarketing.com/playbooks/javascript-seo-making-dynamic-sites-crawlable/ |

The rule of thumb that catches all six is to judge the body. If a request for a content page comes back with something you could not read the article from, the status code attached to it does not matter.

## How to test it, from somewhere that is not your office

You need two things: a request from outside your own network, and a user agent that names a crawler. Both matter, and the first one matters more than people expect.

The test itself is one command, run twice. Once as a crawler, once as a browser, against the same URL.

```
curl -sS -o /dev/null -D - \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://example.com/your-page/

curl -sS -o /dev/null -D - \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36" \
  https://example.com/your-page/
```

Read the status line and the content length on each. Then run it once more without `-o /dev/null` so you can actually see the body, because a 200 carrying a challenge widget is the case the headers will not tell you about. Do this against three URLs at minimum: your homepage, one deep content page, and your robots.txt, since rules are sometimes scoped to paths and a homepage that works proves nothing about the article you want cited.

Use the real user agent strings from each operator's current documentation rather than the one above, which is here to show the shape. The strings change, and a stale one can fail for the wrong reason.

### Why your office network is the one place this cannot be tested

Your office IP has history with your CDN. It is a business or residential address rather than a cloud range. Very likely someone added it to an allowlist at some point so the team would stop locking itself out during deploys. And edge bot scoring weighs all of that alongside the user agent, which means a request from your desk is a different request in every dimension the scoring actually cares about.

So you can set the user agent to GPTBot from your laptop, get a clean 200, and conclude the site is fine while a real crawler from a data center range is collecting 403s. Run the test from a server, a cloud shell, a VPN endpoint, a phone on cellular data, or any host that does not share your reputation. If the two locations disagree, believe the external one.

## Where the setting usually lives

I am going to stay general here on purpose. Provider dashboards get reorganized constantly and a confident set of menu paths in a blog post is a good way to send somebody looking for a screen that was renamed last quarter. What does not change is the category of setting, and there are three of them.

- **Bot management.** Most major CDNs now ship a toggle or a managed category specifically covering AI scrapers and crawlers, and on some plans it defaults to on. This is the first place to look, because it is the one that blocks an entire class of agent in a single switch that reads as obviously sensible to whoever flipped it.
- **The web application firewall.** Managed rule sets plus whatever custom rules your team has written. Custom rules are where the crude matches hide, including the classic rule that fires on the substring bot appearing anywhere in a user agent string. Read your custom rules line by line. Somebody wrote each of them for a reason, and the reason is frequently no longer current.
- **Rate limiting.** Thresholds tuned for human browsing will trip on an agent that requests four pages in three seconds. This one produces intermittent blocking, which is harder to reproduce and easier to dismiss as a fluke.

If you are on a managed host or a platform rather than a CDN you configure yourself, the same three layers exist and some of them may not be exposed to you. Ask your host directly whether AI crawler categories are blocked at their edge by default. It is a fair question and they will have an answer.

One practical note on how to word the request. Marketing asking IT to turn off bot protection is a conversation that ends badly and deserves to. Asking to allow a specific named list of verified agents, while everything else stays exactly as it is, is a much smaller request and a more accurate description of what you want.

## The tradeoff you are actually making

The scraping problem is real. Sites do get hammered by automated traffic that costs money in bandwidth and gives nothing back, and the people who turned on the blocking were usually responding to a real incident. Any version of this advice that pretends otherwise is not worth following.

But the decision is not one switch, and treating it as one is where the value gets lost. The operators run separate agents for separate jobs. Some gather content in bulk to build something. Others fetch a single page because a person asked a question a moment ago and is waiting on the answer. Those two transactions have nothing in common from your side of the exchange. The first takes your content and hands you nothing directly. The second is where your page gets to become a cited source in front of somebody who is actively looking for what you sell.

robots.txt can answer those two questions differently in the same file, agent by agent, and that is the right place to express the policy because it is a policy statement. The edge is a defense mechanism, and it is bad at expressing nuance because it was never built to. Which of the two files does what, and which one does nothing at all, is the subject of llms.txt vs robots.txt: https://www.winstondigitalmarketing.com/playbooks/llms-txt-vs-robots-txt/

Most of the businesses we work with land in the same place once it is laid out: allow retrieval, make a considered call on bulk collection, keep the edge defenses doing their actual job against the traffic that is genuinely abusive. What almost nobody chooses, once they can see the choice, is the category-wide block that refuses both without knowing the difference.

## Confirming the fix, which is not optional

Changing a setting and believing the confirmation dialog is how the original problem got made. Verify in two places.

1. **Immediately, from outside.** Re-run the curl test from the same external host. You want a 200, a content length in the same neighborhood as the browser request, and your actual prose visible in the body. Check the body. A challenge page returns a 200 too.
2. **A few days later, in the edge logs.** Pull the logs, filter to the agents you unblocked, verify the requests are genuine by reverse DNS or the operator's published IP ranges, and group by status code. What you are looking for is real agents getting real 200s, which is the only evidence that the change held for traffic you did not generate yourself.

Edge logs, specifically, for the same reason the problem was invisible. A request refused at the edge never reaches your origin, so in origin logs a blocked agent and an agent that never visited produce identical output, which is to say nothing. Absence and refusal are the two findings you most need to tell apart here. The full method for that pass is in AI crawler log analysis: https://www.winstondigitalmarketing.com/playbooks/ai-crawler-log-analysis/ . Treat the log files as sensitive while you have them, since they carry IP addresses and query strings that can hold personal data.

Then put it on a list. Re-run the external test after any CDN migration, any WAF rule change, any bot protection plan upgrade, and any incident where somebody tightened the rules at two in the morning to stop an attack. Those are the moments this breaks, and none of them are made by people thinking about AI visibility at the time.

## Hiring someone to sort this out

This is a short, cheap piece of work that a lot of GEO providers never touch, because it lives on the infrastructure side of a line that content people do not usually cross.

Ask whether they will test crawler access from outside your network, and listen for whether the answer distinguishes that from testing from anywhere. Ask what response codes they treat as a block, and whether an empty 202 or a 200 carrying a challenge widget is on their list. Ask whether they will read your WAF custom rules or only your robots.txt. And ask how they will confirm the fix, since anyone who answers with the dashboard rather than the logs has not done this before.

On our side the pricing is published instead of quoted after a call, and every engagement is a written scope with the number attached before you pay anything. The structure is on the pricing page: https://www.winstondigitalmarketing.com/pricing/ . The done-for-you version of this work sits inside our generative engine optimization service: https://www.winstondigitalmarketing.com/services/generative-engine-optimization/ , and if you want to know where you stand before talking to anyone, the free audit: https://www.winstondigitalmarketing.com/audit/ runs the same single-page read that turned up the 202 above.

## Frequently asked questions

### Why is my CDN blocking AI crawlers when my robots.txt allows them?

Because robots.txt is a request your origin server publishes, and the CDN answers before the request ever reaches your origin. A bot management rule at the edge inspects the user agent, decides the request looks automated, and returns a challenge or a refusal on its own authority. Nothing about that process consults your robots.txt, so the file can say Allow: GPTBot on every line and still describe a permission that is never exercised. This is almost never a deliberate decision. It is usually a default that was on when the site was set up, or a category rule someone enabled during a scraping incident and never revisited.

### What does an empty 202 response to an AI crawler mean?

It means the edge accepted the request and handed back nothing, which is a bot challenge wearing the clothes of a success. A 202 is in the 2xx family, so uptime monitors count it as fine, and an automated crawler sees a response it cannot use. The content length is near zero or the body is a script tag with no prose in it. Treat any 2xx with an empty or tiny body as a block until you have proven otherwise, because it is the single most expensive failure mode here: it does not appear as an error in any report your team is already reading.

### How do I test whether my site is blocking AI crawlers?

Make a plain HTTP request from outside your own network with the crawler's user agent string, and look at the status code and the size of the body. A single curl does it: request your page with the user agent set to GPTBot or ClaudeBot, then request the same page again with an ordinary browser user agent, and compare the two. If the browser request returns your page and the crawler request returns a 403, a 503, an empty 2xx, or a body that is a fraction of the size, the edge is treating the crawler differently. Run it against your homepage, one deep page, and your robots.txt, since rules are sometimes scoped to paths.

### Why does testing AI crawler access from my office network prove nothing?

Because your office IP is very likely on an allowlist, and because the edge scores the whole request rather than the user agent alone. Your network has a history with the CDN, your requests come from a residential or business range rather than a data center, and any rule your team wrote to stop locking itself out applies to you and not to a crawler coming from cloud infrastructure. A request you make from your desk is a different request in every way that the bot scoring cares about. Test from a machine outside your network, or from any host that does not share your reputation, before you conclude anything.

### Where do I change AI crawler blocking in Cloudflare or another CDN?

Look in three places rather than one, because the block can live in any of them. There is usually a bot management or bot protection area with a toggle covering AI scrapers and crawlers as a category. There is a web application firewall with managed rule sets and any custom rules your team wrote, which is where a rule matching on the word bot in a user agent tends to hide. And there is rate limiting, which trips on the burst pattern a retrieval agent produces when it fetches a few pages in quick succession. The exact menu names change often enough that the reliable move is to search your provider's current documentation for the feature rather than trusting a screenshot from a blog post, including this one.

### Should I block AI crawlers to stop scraping and content theft?

It is a real tradeoff, and the answer is rarely all or nothing. Bulk collection agents gather your content to build something and hand you nothing directly back. Retrieval agents fetch a page because a person asked a question seconds ago and is waiting, which is the moment your page can become a cited source in front of a buyer. Those are different transactions and robots.txt can answer them differently in the same file, agent by agent. Most businesses we work with end up allowing retrieval and making a considered call on collection. What almost nobody wants, once they see it stated plainly, is a category-wide edge rule that refuses both without distinguishing them.

### How do I confirm that unblocking an AI crawler actually worked?

In two places, in this order. First repeat the external curl test and confirm you get a 200 with a full-size body and your actual prose in it. Then go to the edge logs a few days later and look for verified requests from the agents you unblocked, grouped by status code, which is what proves the change held for real traffic rather than for your one test request. Edge logs matter more than origin logs here for the same reason the block was invisible in the first place: a request refused at the edge never reaches your origin, so in origin logs a blocked agent and an absent agent look identical.