# Crawl Budget: When It Matters and What to Do About It

**Author:** John Morabito (Founder, /winston)
**Published:** September 19, 2026
**Reading time:** 9 minutes
**Canonical:** https://www.winstondigitalmarketing.com/playbooks/crawl-budget-optimization/

Most of the sites that ask me about crawl budget do not have a crawl budget problem. That is the honest first answer and it saves people a quarter of wasted effort. A few hundred pages on ordinary hosting, Google fetches all of them within a few days, and whatever is keeping a page out of the results has nothing to do with how many requests the crawler was willing to spend. The phrase comes up in technical SEO conversations far more often than the problem turns up in the sites being discussed.

Then there is the other kind of site. A catalog where filters multiply into millions of addresses. A publisher with twenty years of archive and a tag system that cross-cuts all of it. A CMS that mints a fresh URL every time someone sorts a table by a different column. On those sites crawl budget is real, it is expensive, and it is one of the few technical problems where engineering time pays back. The whole job is telling which kind of site you are looking at before you start work.

## What the phrase covers

Google describes crawl budget as two things multiplied together, and the split is worth keeping because the two halves have different fixes. The first is crawl capacity: how many simultaneous connections Google is willing to open and how long it waits between them, adjusted continuously based on how your server responds. Fast responses and clean status codes raise it. Slow responses and server errors lower it, quickly, because the crawler is explicitly built to avoid knocking your site over.

The second is crawl demand: how much Google actually wants to fetch from you. That is driven by how many URLs it knows about, how popular they are, and how often they change in ways that turned out to be worth re-fetching. A site with a hundred thousand URLs that never change and that nobody links to will not get crawled hard no matter how fast the server is.

A crawl budget problem is when demand is being spent on the wrong URLs, or when capacity is low enough that the crawler cannot get through the URLs that matter in a reasonable window. Those two want different fixes, and confusing them is why most crawl budget projects produce nothing.

## Who actually has the problem

Two situations produce a genuine one. The first is scale, which in practice means a site where the number of crawlable addresses runs well ahead of the number of pages anyone would want: large ecommerce catalogs with faceted navigation, marketplaces, job and listing sites, classifieds, big publisher archives, anything with a parameter system that combines. The count that matters is not how many products or articles you have. It is how many distinct URLs a crawler can reach by following links and permuting parameters, which on a filtered catalog is a number nobody on the team has ever actually calculated.

The second is an event that floods the queue. A migration that changes every URL on the site gives the crawler two versions of everything to work through. A CMS or plugin change that starts appending a tracking parameter to internal links can double the known URL set in a week. An accidental infinite space, most often a calendar widget that will happily generate next month forever, can add more URLs than the real site has in a matter of days. These are the cases where a site that never had a crawl problem suddenly has one, and the trigger is usually datable to a specific deploy.

If you are not in one of those two situations, stop reading about crawl budget and go fix something that will move the number. For a small site, the reasons pages do not rank are almost always the ordinary ones.

## How to tell whether you have it

This is the step people skip, and skipping it is how crawl budget work turns into theater. There are two places to look and they answer different questions.

### Search Console crawl stats

The crawl stats report, under Settings rather than anywhere obvious in the main navigation, shows total crawl requests over the last ninety days, average response time, and a set of breakdowns. The useful ones are by response code, by file type, by Googlebot type, and by purpose, where purpose splits into discovery of new URLs and refresh of ones Google already has. Read it for shape. There is no total that counts as good, so the number on its own tells you nothing.

The patterns that mean something: average response time trending up while total requests trend down, which is capacity being throttled because your server got slower. A large and rising share of requests returning 404 or 301, which is crawl being spent on addresses that go nowhere. A refresh share that dwarfs discovery on a site that publishes constantly, which means Google is busy re-checking old URLs instead of finding your new ones.

### Server logs

Crawl stats tells you the totals. Logs tell you which URLs, and that is the part you need to make a decision. Pull a month of access logs, filter to verified crawler traffic, and group the requests by URL pattern rather than by URL. The question you are answering is what share of the crawl is going to addresses you would not care if Google never saw again: filter combinations, internal search results, session or tracking parameters, paginated tails nobody links to. On a site with a real problem, that share is not subtle. The mechanics of doing this, including verifying that the requests came from who they claim, are in [log file analysis for SEO](https://www.winstondigitalmarketing.com/playbooks/log-file-analysis-for-seo/), which is the diagnostic half of this playbook.

One more signal sits in the page indexing report: a large and growing pile of URLs sitting in the status for discovered but not yet crawled. That status means Google knows the address exists and has chosen not to spend a request on it yet. A handful is normal. Thousands, on a site that is otherwise healthy, is Google telling you it has better things to do with your crawl than the URLs you just submitted.

## What eats it

Every real case I have worked on came down to some combination of six things, roughly in this order of frequency.

- **Parameter and facet explosion.** Filters that combine freely produce a URL space that grows multiplicatively while the page count grows not at all. Eight filters with five values each is not forty pages, and the crawler has no way to know the combinations are worthless unless you tell it. This is the single biggest crawl consumer on large ecommerce sites and it has its own playbook in [faceted navigation SEO](https://www.winstondigitalmarketing.com/playbooks/faceted-navigation-seo/).
- **Soft 404s.** A URL that returns 200 with a page that says no results found, or an out-of-stock product that renders an empty shell. The crawler was told the page exists and is fine, so it keeps coming back. A filter combination that matches nothing is the most common source, and it is the worst one, because the same template generates an unlimited number of them.
- **Redirect chains.** Every hop is a separate request. A chain of four redirects costs four times what the destination should have cost, and after enough hops the crawler gives up before arriving. Chains pile up across migrations, each new rule pointing at the previous rule instead of at the live URL.
- **Near-identical pages.** Location pages differing by a city name, product variants with their own URLs and the same description, tag archives that list the same twelve posts in a different order. The crawler fetches all of them and gradually learns that re-fetching your URLs is not worth much, which is a demand problem that follows you around. The cleanup is its own project, covered in [fixing thin and duplicate content](https://www.winstondigitalmarketing.com/playbooks/fixing-thin-and-duplicate-content/).
- **Slow server response.** The one lever that works directly on capacity. If your time to first byte is bad under crawler load, Google throttles, and then nothing downstream matters because the requests are not being made at all. This is a server and caching problem, measured at the origin, and it has almost nothing to do with how heavy the page is in a browser.
- **Pagination that hides the tail.** Deep pages in a paginated series reachable only through a long click chain, or through links a crawler cannot follow, mean the items on them are effectively invisible while the crawler still spends requests working down the series. The handling that keeps deep items reachable is in [pagination SEO best practices](https://www.winstondigitalmarketing.com/playbooks/pagination-seo-best-practices/).

## The fixes, in the order they pay

Assuming the diagnosis came back positive, this is the sequence I work in. The order matters, because the first two determine whether the rest of the work is even visible.

- **Fix response time at the origin.** Caching, database queries, and whatever the slowest template is doing. Capacity responds to this within days and it lifts everything else.
- **Stop generating the URLs.** The most effective crawl budget fix is usually upstream of SEO entirely: a filter UI that uses POST or fragments for the combinations nobody searches for, internal links that do not carry tracking parameters, a calendar that stops rendering links past a reasonable horizon. A URL that never gets created never needs to be managed.
- **Consolidate what is legitimately duplicate.** Canonical tags on variants and sort orders that show the same items. This is a demand fix rather than a capacity fix, and it works slowly, because the crawler still fetches the URL to read the tag.
- **Disallow the patterns that are genuinely worthless.** Robots.txt is the only tool that stops the request from happening at all, which makes it the only true crawl saver on the list. It comes with a trap: a URL you disallow can still be indexed if something links to it, and the disallow prevents Google from ever seeing a noindex tag on that page. Block what should never be fetched. Use noindex for what needs to be fetched and then dropped.
- **Flatten the chains.** Rewrite every redirect to point at the current destination in one hop. There is no judgment involved, and on a site with a few migrations behind it this is often the biggest single-day cut in wasted requests available.
- **Point internal links at what you want crawled.** Crawl follows links, so the fastest way to change where it goes is to change where your own site sends it. Link the pages that earn money like they earn money, and stop feeding the crawler into the sprawl through faceted link blocks and archive widgets.
- **Keep the sitemap honest.** Canonical, indexable URLs only, with accurate lastmod values. A sitemap full of redirects and 404s, or one where every lastmod updates nightly whether or not anything changed, teaches the crawler to discount the file.

## What is not worth your time

A few things have enough folklore attached that they are worth naming. The crawl-delay directive in robots.txt is not supported by Google, so putting it there does nothing to Googlebot regardless of what it does to other crawlers. The URL parameters tool that used to live in Search Console was retired in 2022, so parameter handling is now a job for your own architecture and robots rules. Adding noindex to a page does not save crawl, because the page has to be fetched for the directive to be read. And nofollow on internal links is a hint about the link rather than a fence around a URL space.

The largest waste, though, is a small site running any of this at all. If your site has a few thousand URLs and your server answers quickly, Google is crawling everything you have and the reason a page is not ranking is sitting somewhere in content, links, or relevance. Crawl budget is a scale problem, and adopting a scale problem you do not have is a way to look busy while the actual issue goes unexamined.

## Putting it together

Confirm the problem before you work on it, using crawl stats for the shape and logs for the detail. Separate the capacity half from the demand half, because response time and URL sprawl are different jobs. Then work the fixes in order: speed at the origin, stop minting URLs, consolidate, disallow, flatten, relink, and keep the sitemap clean. On a site that genuinely has the problem, the crawl goes back to the pages that earn something and new work starts getting indexed faster. There is no crawl budget score to screenshot, which is part of why this work keeps getting skipped.

This kind of architectural cleanup is core to what we do in our [SEO service](https://www.winstondigitalmarketing.com/services/seo/), and if you want a read on whether your site has the problem at all, the [free AI visibility audit](https://www.winstondigitalmarketing.com/audit/) starts with how your site is being crawled and read today.

## Frequently asked questions

### What is crawl budget in SEO?

Crawl budget is the number of URLs a search engine is willing and able to fetch from your site in a given period. Google describes it as two factors combined. The first is crawl capacity, meaning how many requests your server can take without slowing down or erroring, which the crawler adjusts continuously based on how you respond. The second is crawl demand, meaning how much the engine actually wants to fetch from you, driven by how many URLs it knows about, how popular they are, and whether re-fetching them has historically been worth it. You have a problem when that budget is being spent on URLs you do not care about, or when capacity is low enough that the pages you do care about wait.

### Does my small business site have a crawl budget problem?

Almost certainly not. If your site has a few thousand URLs or fewer and your server responds quickly, search engines are fetching everything you have published and the reason a page is not ranking is somewhere else entirely. Crawl budget becomes a real constraint at scale, meaning large catalogs with faceted filters, marketplaces, listing sites, and deep publisher archives, or after an event that floods the crawl queue such as a migration or a CMS change that starts generating unlimited URLs. Adopting the problem when you do not have it is a reliable way to spend a quarter on work that changes nothing.

### How do I check my crawl budget in Google Search Console?

The crawl stats report lives under Settings rather than in the main navigation, and it shows total crawl requests over the last ninety days, average response time, and breakdowns by response code, file type, Googlebot type, and purpose. Read it for shape. No specific total counts as good. The patterns worth acting on are response time rising while total requests fall, which means capacity is being throttled, and a large share of requests returning 404 or 301, which means crawl is going to addresses that lead nowhere. For which specific URLs are consuming the crawl, you need server logs, because Search Console gives you totals rather than a list.

### Does blocking URLs in robots.txt save crawl budget?

Yes, and it is the only tool on the list that does so directly, because a disallowed URL is never requested at all. Canonical tags and noindex both require the page to be fetched before the directive can be read, so neither saves a request. The trap with robots.txt is that a blocked URL can still end up in the index if other pages link to it, usually with no description, and blocking it is exactly what prevents the engine from seeing a noindex tag you may have placed on the page. The rule of thumb is to disallow what should never be fetched, and to use noindex for what needs to be fetched and then kept out of results.

### Do redirect chains waste crawl budget?

Every hop in a chain is a separate request, so a URL that redirects four times before arriving costs four times what the destination alone would have cost. Past a certain number of hops the crawler stops following and never reaches the page at all. Chains accumulate quietly, usually across successive migrations where each new redirect points at the previous redirect rather than at the current live URL. Rewriting every rule to point directly at the final destination is mechanical work with no judgment involved, and on a site with several migrations behind it, it is often the biggest single-day reduction in wasted crawl available.