Faceted Navigation Is Eating Your Crawl Budget

Open Google Search Console. Go to Pages. Look at the number under "Not indexed."

If you run a catalog of any size, that number is probably larger than the number under "Indexed," and considerably larger than the number of products you actually sell.

Where did the rest come from?

They came from your filters. Every colour, every size, every price bracket a shopper can click, in every possible combination, each one generating a URL that Googlebot will happily crawl until it runs out of patience and leaves.

The Arithmetic Nobody Runs Before Launch

Say your skincare category has five filters. Brand, skin type, size, price range, and sort order. Each has a handful of options.

Ten brands. Four skin types. Three sizes. Five price brackets. Four sort options.

Multiply them and you have 2,400 unique URLs from a single category page. Add a second sort parameter and it doubles. Allow filters to be applied in any order, which most platforms do by default, and the permutations multiply again because ?colour=blue&size=50ml and ?size=50ml&colour=blue are, to a crawler, two different pages showing identical content.

Now do that across forty categories.

That last URL deserves a moment. Page 47 of an empty filter combination. It contains no products. It will never rank for anything. Googlebot crawled it, parsed it, decided it was worthless, and in doing so spent a fraction of the finite attention it allocates to your domain.

Multiply that fraction by four hundred thousand and you have a crawl budget problem.

What Crawl Budget Actually Is, Without the Mysticism

Google allocates each domain a rough quota of crawling: how many URLs it will fetch, how often it returns, how deep it goes.

The quota is not published, and it is not fixed. It responds to your server's speed, your site's historical value, how often content changes, and how much of what Googlebot fetched last time turned out to be worth having.

That last factor is the one brands never account for.

If ninety percent of what Googlebot crawls on your domain is duplicate filter permutations, it learns that your domain is mostly noise. It crawls less. It returns less often. Your new product launches sit unindexed for weeks while the crawler works through parameter combinations of a category you discontinued last year.

The Second Problem: Duplicate Content at Industrial Scale

Crawl waste is the visible cost. Signal dilution is the expensive one.

Suppose your ceramide moisturizer appears on the main category page, on the filtered page for blue packaging, on the filtered page for 50ml sizes, and on the filtered page for that brand. Four URLs, near-identical content, each accumulating a trickle of internal link equity.

Google now has to decide which of the four deserves to rank. It picks one, more or less arbitrarily, and the ranking signals you intended to concentrate on your primary category page are instead spread across four pages that individually rank for nothing.

This is the mechanism by which a well-built site with genuinely good content fails to rank for its most obvious commercial term.

Product variants make it worse

One shirt in five colours. Five URLs. Five identical descriptions, because nobody writes five hundred words about the difference between navy and midnight blue.

Google reads five thin pages, not one strong one, and treats the whole cluster with suspicion.

How to Fix It Without Breaking the Shopping Experience

The instinct is to block everything. That instinct is wrong, because filters exist for a reason and some of them attract genuine search demand.

"Ceramide moisturizer for oily skin" is a real query with real volume. If that combination is only reachable as a filtered URL, blocking it forfeits the traffic.

So the work is triage, not demolition.

The out-of-stock question everyone gets wrong

A product sells out. The obvious move is to remove the page, or return a 404.

Both destroy every backlink that page ever accumulated, along with whatever ranking authority it built over eighteen months.

Keep it live. Return a 200. State clearly that it is out of stock. Add a restock notification form, and link aggressively to alternatives that are available. The page keeps its equity, the shopper keeps a path forward, and when inventory returns you rank immediately rather than starting over.

For products permanently discontinued, redirect with a 301 to the closest parent category. The equity transfers. A 404 throws it away.

What Category Pages Are Actually For

Here is the strategic point underneath all the technical detail.

Google favours category pages for generic commercial queries. Search "running shoes" and you get collection pages, not individual product listings, because the searcher wants to browse and compare rather than land on one item.

Category pages are therefore your highest-value organic asset, and they are the pages most damaged by faceted navigation chaos.

Category pages enriched with genuinely unique content, written for shoppers rather than copied from a manufacturer spec sheet, rank up to 2.7 times higher than those relying on filter text and product names alone. That multiplier is the entire argument for treating category pages as editorial properties rather than as database queries with a template wrapped around them.

But a category page cannot rank if its authority is scattered across two thousand filtered variants of itself.

Where This Connects to Everything Else

Crawl budget governance is unglamorous work. It produces no impressive slide for a quarterly review. Nobody has ever been promoted for consolidating canonical tags.

It also determines the ceiling on everything else you do.

Content strategy assumes Googlebot reaches your content. Schema deployment assumes the page gets crawled. AI engines, which increasingly decide what shoppers even consider buying, rely on search crawlers to access your catalog in the first place. A store Google cannot index properly will not get cited well by ChatGPT either, however good the structured data.

That is worth sitting with, because it inverts the usual priority order. Generative Engine Optimization for e-commerce is not an alternative to technical SEO. It sits on top of it, and it collapses without it.

I wrote about this dependency at some length in Found Before They Search, because Indonesian brands were investing heavily in AI visibility while their category pages fought two thousand filtered clones of themselves for the right to be indexed. Both matter. One comes first.

The companion volume, Cited or Silent, covers the AI citation side in depth. Both are free as gated editions, and both are published in paperback and hardcover and listed on Google Play Books and Apple Books.

Where to Start Tomorrow Morning

Run a crawl of your own site with any standard tool. Sort the discovered URLs by parameter count.

Count how many contain two or more parameters. Count how many return a page with fewer than three products on it.

That second number is what your competitors are not wasting Googlebot's time on. Everything in ecommerce SEO gets easier once it approaches zero.

Frequently Asked Questions

What is crawl budget, in practical terms?

It is the rough quota of URLs Google will fetch from your domain within a given period. It is not published and not fixed. It responds to server speed, historical site value, content freshness, and crucially how much of what Googlebot previously fetched turned out to be worth indexing. Sites that serve mostly duplicate filter permutations get crawled less over time.

Should we block all filtered URLs in robots.txt?

No. Some filter combinations match real search demand, such as skin type or product function, and blocking them forfeits that traffic. The work is triage: give demand-matching filters clean static URLs, canonicalise the rest back to the parent category, and block only the impossible combinations such as empty filters and deep pagination.

How do we handle out-of-stock product pages?

Keep them live with a 200 status. Deleting the page or returning a 404 destroys every backlink and all the ranking authority it accumulated. State clearly that the item is out of stock, add a restock notification form, and link to available alternatives. For permanently discontinued products, use a 301 redirect to the closest parent category so the equity transfers rather than evaporating.

Why do product variants cause duplicate content problems?

Because five colour variants typically generate five URLs with near-identical descriptions. Google reads five thin pages rather than one strong one, splits ranking signals across them, and treats the cluster with suspicion. Either canonicalise all variants to a master URL, or write genuinely distinct content for each.

Does parameter order really create duplicate URLs?

Yes, and it is one of the largest sources of unnecessary crawl waste. To a crawler, a URL with colour before size is a different page from one with size before colour, even though both render identical content. Forcing your platform to output parameters in a consistent sequence can eliminate more than half the permutations on a typical catalog.

How much better do category pages rank with unique content?

Up to 2.7 times higher than category pages relying on filter text and product names alone. Google favours category pages for generic commercial queries because searchers want to browse and compare, which makes them the highest-value organic asset on most e-commerce sites, and also the pages most damaged by faceted navigation problems.

---