If the old logic stayed in place, the technical URL tail would keep growing with the catalog. But blocking every filter would remove pages with real search demand. The solution had to separate search landings, navigation states, and purely technical combinations.
Implemented · measured
Faceted navigation: rules instead of millions of excess URLs
Google already knew about 65M+ non-indexed URLs, and crawling one large category reached roughly 5M addresses. We changed not only indexability but the URL model itself: justified search landings received clean stable URLs instead of parameter combinations, while the sitemap stopped reflecting the technical tail of the catalog.
Case at a glanceDiagnosed the source of URL growth, designed page-type rules, the target URL model and exception policy, aligned requirements and validated implementation.
Non-indexed URLs known to Google fell from 65M+ to about 1.3M. The indexed footprint consolidated from roughly 500K to 181K pages. The sitemap shrank from 146 files with 50K URLs each to 4 files, and justified filter landings received clean URLs instead of parameter combinations.
65M+Before changes
~1.3MAfter changes
Non-indexed URLs known to Google before and after the architecture change.
01Filter state→02Demand & intent→03SEO policy + clean URL→04Crawl / index rules→05Sitemap
The solution connects page generation, URL format, indexation rules and the sitemap. That is why the outcome is measured in tens of millions of URLs rather than one local cleanup.
The rule that turns a URL into a search page was redesigned
The problem started in page generation and then propagated into crawling, indexation, Search Console and the sitemap. The solution therefore changed several layers at once: which combinations are allowed to become pages, which URL is canonical for a landing, how SEO policy is assigned and which URLs are declared through the sitemap. Clean URLs and the 146 → 4 sitemap reduction are consequences of one architecture, not isolated cleanups.
Key decision07
Default policy plus manual exceptions
Scaling required predictable defaults without removing SEO ownership of important exceptions.
Default behavior
Each page type receives an agreed SEO policy.
Exception path
The tag-page module supports justified manual overrides for non-standard cases.
One category exposed the scale of the problem
In one large category, the technically reachable URL space grew to roughly five million open URLs while the number of genuinely useful search landings was orders of magnitude smaller.
Why blocking all filters was the wrong answer
Different page types served different jobs.
Search landing
Justified when there is distinct demand and intent.
Navigation page
Can still help shoppers even without meaningful standalone search volume.
Technical combination
Should not become a search entity merely because a URL can be generated.
URL model: parameters → clean stable URL
Pages that genuinely earned the right to become search landings received a different URL model.
Before
The URL reflected a technical filter combination and depended on parameter logic from the interface.
After
A justified landing gets a stable human-readable URL and its own SEO behavior. Technical combinations do not receive that status automatically.
How a filter earned the right to become a page
The decision was not driven by a single signal.
Demand
Check for a distinct query group.
Intent
Verify a distinct user task.
SERP review
Compare with the documents already serving the demand.
Cannibalization
Avoid a new page that competes with an existing landing.
Technical policy
Then define indexability, canonical, crawl policy and URL format.
The sitemap began to represent the target index
After the URL architecture cleanup, the sitemap stopped mirroring almost everything the catalog could technically generate.
146 files
the historical sitemap was split across 146 files with 50K URLs each
4 files
after cleanup, only a compact set for target indexable pages remained
Why it matters
the sitemap became a declaration of the intended search layer rather than a reflection of technical URL noise
What changed in the architecture
The reason a URL is allowed to become a search landing changed.
Before
A combination exists in the interface → a URL appears → it can leak into the technical search tail.
After
Page type → demand and differentiation check → SEO policy → clean URL and search entity only when justified.
My responsibility vs engineering
I owned the SEO architecture and acceptance criteria; engineering implemented the platform behavior.
My responsibility
Root-cause diagnosis, search-entity model, landing criteria, URL architecture, sitemap requirements, exceptions and UAT.
Engineering
Catalog logic, clean URL implementation, sitemap changes and platform behavior.
Context & measurement
- Context
- Large e-commerce catalog · production work
- Scale / period
- 65M+ → ~1.3M non-indexed URLs · ~500K → 181K indexed pages · 146 → 4 sitemap files
How it was measured
The case uses observable technical outcomes: non-indexed URLs known to Google, indexed footprint and sitemap-file count after consolidation.
Catalog architecture
Facing a similar challenge?
Tell me where SEO is blocked by product, engineering or data.