Google doesn't crawl every URL on a site as fast as it appears. It allocates a crawl budget — roughly, the number of URLs Googlebot is willing and able to fetch from a given site within a given timeframe — and that budget is finite. For a small site with a few hundred pages, this is rarely a constraint. For a site with millions of URLs (large e-commerce catalogs, classifieds, forums with infinite thread pagination), crawl budget becomes a real bottleneck: pages you want indexed can sit uncrawled for weeks while the budget gets consumed elsewhere.

How Google decides how much to crawl

Crawl budget is shaped by two main forces. Crawl capacity limit is about what your server can handle — Google throttles its crawl rate based on how your server responds; slow response times and 5xx errors cause Google to back off and crawl less, not more. Crawl demand is about perceived value — URLs with more inbound links, more traffic, or higher-quality signals get crawled more often, while low-value or rarely-changing URLs get deprioritized. Faster, more reliable servers combined with genuinely valuable pages increase effective crawl budget without any direct 'budget' setting to tune.

What increases vs. decreases crawl budget

Factor	Effect on crawl budget	Why
Fast, stable server response times	Increases	Google crawls more aggressively when it can without straining the server
High rate of 5xx errors or timeouts	Decreases	Google backs off to avoid overloading a struggling server
Large volumes of low-value duplicate URLs	Decreases effective budget	Budget gets consumed on pages not worth indexing
Strong internal linking and external links to key pages	Increases demand for those pages specifically	Signals the page is worth prioritizing
Frequent legitimate content updates	Increases recrawl frequency	Signals the page is worth checking again soon

Identifying crawl waste

Crawl waste is budget spent on URLs that don't deserve it: infinite faceted-navigation combinations (every permutation of color, size, and price filter generating a unique crawlable URL), internal search result pages, calendar/date-archive pages generating endless future or past dates, session-ID or tracking-parameter duplicates, and soft-404 pages that return 200 but have no real content. The fastest way to find it is comparing a crawled-URL list from log files against the list of URLs you actually want indexed — anything crawled heavily that isn't on the second list is a waste candidate.

crawl-waste-examples.txt

text

/shoes?color=red&size=9&sort=price_asc&page=4   # faceted combinatorial explosion
/search?q=running+shoes                          # internal search results page
/events?month=2031-03                             # far-future calendar page, no real content
/products/item-882?sessionid=8827abf              # duplicate via session parameter

robots.txt and parameter handling to focus crawling

Once waste patterns are identified, robots.txt disallow rules are the primary lever to stop crawling them outright — effective for entire path patterns like internal search or session-parameterized URLs. For URL parameters that create duplicate but still-useful content (like ?sort= on a category page), the better fix is usually a canonical tag rather than a full disallow, since disallow blocks the crawl entirely and prevents Google from even seeing the canonical signal. Google Search Console previously offered a URL Parameters tool for this; it's since been retired, so parameter handling today relies on consistent internal linking, canonicalization, and robots.txt rather than a dedicated parameter-configuration UI.

robots.txt

text

User-agent: *
Disallow: /search
Disallow: /*?sessionid=
Disallow: /*?*sort=*&*page=        # block deep combinatorial facet crawls
Allow: /

Sitemap: https://example.com/sitemap-index.xml

Disallow stops crawling, not indexing of already-known URLs

The same trap as blocking already-indexed pages applies here: disallowing a URL pattern in robots.txt stops future crawling, but a URL that's already indexed can persist in the index without a description if it's not also handled with noindex before being blocked. For genuinely unwanted but already-indexed waste, let it be recrawled with noindex first, then block it afterward if needed.

Prioritizing what should get crawled

Beyond blocking waste, crawl budget optimization is also about actively directing attention: a clean, shallow internal link structure, a sitemap limited to genuinely canonical URLs, pruning or consolidating thin/low-value pages instead of letting them accumulate, and fixing server errors that cause Google to throttle crawl rate defensively. On large sites, these changes compound — removing a few hundred thousand low-value combinatorial URLs can measurably speed up how quickly genuinely new or updated pages get discovered.

Crawl budget mostly matters at scale

For sites under roughly 10,000 pages, crawl budget is rarely the binding constraint on indexing — Google can usually crawl a site of that size thoroughly within its normal allocation. Chasing crawl budget optimization on a small site is often solving a problem that doesn't exist yet; the ROI shows up clearly once a site crosses into the hundreds of thousands of URLs.

What's next

Everything in this lesson is theoretical without a way to verify what's actually happening — log files are the one source of truth for what Googlebot really crawled, how often, and where the budget actually went.

Next: Log File Analysis for SEO →

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →