You’re mapping a messy source vocabulary onto a clean target taxonomy. The target has about 5,600 slots, so the job must be roughly 5,600 decisions — right? The backlog is already past 8,900, and still climbing.
Table of Contents
The instinct, and why it under-counts
You have a messy source vocabulary to map onto a clean target taxonomy: store category breadcrumbs onto a standard product taxonomy, free-text tags onto a controlled list, legacy codes onto a new scheme. The target is finite and known — you can SELECT count(*) it. So the work feels bounded by it: one decision per target node, a few thousand calls, done.
That estimate can be off by multiples, because the relationship is almost always many-to-one: many distinct source items correctly map to the same target node. “Running Shoes,” “trail runners,” and “athletic footwear — men’s” can all be the same destination. The target bounds the destinations; it says nothing about the volume of things arriving at them. And that volume is a property of your source — how much of each catalog you’ve crawled and enriched — which keeps growing.
A finite target taxonomy bounds where items land, never how many items there are to route.
The concrete case
A price-comparison engine maps each store’s breadcrumb category — a “chain” like one marketplace’s Entry Doors or another store’s Oil Filler Caps — onto the Google Product Taxonomy. Someone reasonably estimated the backlog at “around 5,000,” close to the taxonomy’s size, and asked whether the whole job was therefore bounded by it.
The data said otherwise:
| Quantity | Count |
|---|---|
| Target taxonomy nodes | 5,595 (4,719 leaves) |
| Source chains already mapped | 8,959 — already past the taxonomy |
| …onto how many distinct target nodes | 2,388 |
| Chains per node | 3.75 average, up to 39 |
One node — Art & Crafting Materials > … > Fabric — holds 39 distinct store chains. Pants holds 36. If the job were taxonomy-bounded you’d expect about one chain per node; instead it’s 3.75 and climbing, and more than 8,900 chains have already been mapped onto fewer than 2,400 nodes. Counting the ~2,930 still unjudged, the real chain universe is ~12,000 and growing — because it tracks how much of each store’s catalog has been crawled and enriched, not the fixed size of the taxonomy.
And the “~5,000” that started it all? A partial view — one recent batch — whose nearness to the taxonomy size was pure coincidence.
Why the wrong estimate is seductive
Three things make the taxonomy-bounded guess feel right:
- The target is the visible, finite thing. You can count the taxonomy in one query. The source volume is implicit in crawl and enrichment state you may not have in front of you — so the number you can see becomes the number you reach for.
- A coincidental near-match feels like confirmation. “~5,000 ≈ ~5,600” looks like a mechanism. But a partial count can land anywhere; one that happens to land near the taxonomy size invites a false explanation. The visible figure isn’t the governing one.
- Many-to-one is easy to forget once you’ve been picturing a tidy one-to-one “category → category” model. The fan-in is real, large, and invisible until you measure it.
A partial count that lands near the taxonomy size isn’t confirmation — it’s a coincidence wearing a mechanism’s clothes.
The fix: size by source, instrument the real driver
The rule is a single substitution in how you estimate. In a many-to-one mapping the source count routinely exceeds the target and keeps growing with coverage, so:
Size a mapping backlog by distinct source keys, not target categories: the taxonomy bounds where items land, not how many there are.
Two practical consequences follow:
- Plan for rounds, not a one-shot. A backlog that scales with ingestion is never “finished” — it’s drained in waves as new source data arrives. That’s the same posture as reserving capacity for a taxonomy that grows with your catalog: design for the stream, not a single terminal pass.
- Instrument the real driver. Track distinct source keys and an enrichment watermark, not progress against the taxonomy. Measured against the target, you’ll read “99% of nodes covered” and feel nearly done — while most of the source work is still ahead of you, because coverage, not the taxonomy, is what’s actually being consumed. A lean, well-sized target taxonomy is still worth having; it just isn’t the yardstick for how much mapping remains.
The short version
In a many-to-one mapping, many source items correctly land on the same target node, so the target taxonomy bounds the destinations, not the volume of work. Sizing a mapping backlog by the number of target categories under-counts it by multiples — in one measured case, 8,959 source chains were already mapped onto just 2,388 of ~5,600 nodes (3.75 per node, up to 39), with ~12,000 chains total and growing. The real driver is source coverage, which keeps expanding as you crawl and ingest. Size the backlog by distinct source keys and an ingestion watermark, not by target nodes; expect the taxonomy to read “fully covered” long before the source work is done; and plan to drain the backlog in rounds, because it scales with ingestion rather than terminating at the taxonomy’s edge.
Frequently Asked Questions
Why isn’t the target taxonomy size a good estimate of the mapping work?
Because the mapping is many-to-one: many source items legitimately share one target node. The taxonomy bounds how many destinations exist, not how many items you must route to them. The item count comes from your source coverage, which is usually larger than the taxonomy and grows over time — so the taxonomy size systematically under-counts the work.
What should I size the backlog by instead?
Distinct source keys — the unique things you have to classify — plus a watermark for how much of the source you’ve ingested so far. That number reflects the actual decisions to make, and it updates as crawling or ingestion brings in more source data. The taxonomy count tells you the shape of the destination, not the volume of the work.
Our partial count came out near the taxonomy size — doesn’t that confirm the bound?
Almost certainly a coincidence. A partial count (one batch, one time window) can land anywhere, and one landing near the taxonomy size invites a false mechanism. Check the full picture: count distinct source keys already mapped and how many land per node. If it’s well above one per node, you have a many-to-one fan-in and the taxonomy size is not your bound.
How do I know when the mapping is “done”?
In a coverage-driven backlog, “done” is relative to an ingestion watermark, not the taxonomy. It’s done for the source you’ve ingested so far; new ingestion reopens it. Track completion as “all distinct source keys up to watermark X are mapped,” and expect to run further rounds as the watermark advances.
Does this apply beyond product taxonomies?
Yes — it’s any many-to-one mapping of a growing source onto a fixed target: tags onto a controlled vocabulary, free-text onto a code list, records onto a reference scheme. Whenever many source values can share one target and the source set grows independently, size the work by the source, and treat the target’s size as a fact about destinations only.
