The report said 80 duplicate groups still to resolve. You’d been resolving them for days. You run it again after a big batch and it still says 80-ish. The backlog isn’t stubborn — the scan is counting work you already finished.
Table of Contents
Why a resolved item still looks like a collision
The production resolver and your quick scan compute groups the same way — normalise each record to a key, bucket by key — but they diverge on one thing the resolver does and the scan forgets: the resolver excludes items it has already placed (members of a confirmed cluster, rows with a frozen decision) before it looks for collisions. A folded duplicate typically keeps its raw fields; it just carries a pointer to its canonical. So by raw key it is still identical to its twin — still a “collision” — even though it is fully resolved. The resolver knows to ignore it; your from-scratch scan doesn’t, so it rebuilds the pile you already cleared.
The tell is a backlog number that won’t go down no matter how much you resolve, or one that’s a suspiciously round multiple of the real remainder. (It’s the measurement that’s wrong, not the data — the same reflex worth having whenever a count looks off: is it a real loss or a stale read?)
Subtract the already-resolved set before you group — exactly as the resolver already does.
The rule: resolved-set minus, every time
Mirror exactly what the resolver does:
- Load the resolved set. The ids already in a confirmed cluster or carrying a decision — one cheap query against whatever table records membership/verdicts.
- Subtract it.
work = [r for r in records if r.id not in resolved and r.has_identity]— also drop anything structurally unresolvable (no stable identity, held for another reason). - Group the remainder. Only now compute collision keys and keep groups with two or more surviving members.
Everything you report — counts, worklists, “providers still to do” — comes off work, never the raw table. The same trap has a cousin: before you start a resolution pass at all, check how much is already resolved. We opened one “fresh” pass that turned out to be 199 groups already confirmed and 324 already proposed — most of the “work” was done, and a raw scan was re-counting it.
Watch it bite in layers
A backlog that won’t fall no matter how much you resolve is the symptom — you’re re-counting finished work.
It compounds when resolution happens in stages. Resolve round one (cluster the easy duplicates); a scan for round two that ignores round one’s clusters re-lists them; you “resolve” them again — recreating what already exists — now fighting a uniqueness constraint or creating a second cluster for the same item. We hit the same miscount three times in one project — each time because a scan grouped raw records without subtracting the clusters a previous stage had already created. Once the subtraction was in, an “80-group” remainder collapsed to 31, and a whole category (“36 still to do”) went to zero: all 36 were already clustered.
When the resolved-set itself is incomplete
Subtracting the resolved set only works if the set is complete. There’s a quieter way to get this wrong: the subtraction is right there in your code, but the thing you subtract from has a hole in it.
Picture the resolver split in two — a generator that scans the source and emits the next batch of un-resolved work, and a processor that judges each item and records the outcome. The generator avoids re-emitting finished work by consulting a resume ledger: the set of keys already handled. With one processor this is airtight — it writes the ledger as it goes, so the generator’s “done” set is exactly right.
Then a second processor appears. It writes the outcome tables perfectly — the item is clustered, decided, authoritative — but it never touches the resume ledger the generator actually reads. Now the generator is blind to everything the second writer did. It faithfully re-emits all of it, and you are back to re-counting finished work — not because the subtraction is missing, but because the set you subtracted from was missing rows.
Being right in the system of record doesn’t help if the generator keys on a ledger the record never updated.
The tell is exact, and worth checking by hand: take a freshly-processed key and look for it in the place the generator will look next. If it’s live in the authoritative table yet absent from every resume ledger, you’ve found it. The fix is one of two rules — pick one and make it the invariant: every writer updates the shared resume state, or the generator derives “done” from the authoritative outcome instead of a parallel ledger (one source of truth, no second writer to forget). The second is the sturdier long-term design; the first is cheaper and often enough. Either way the lesson is the same as the rest of this article — what you subtract must be the complete set of what’s done, not just the part one writer happened to record.
The backstop that catches you anyway
Make the mistake impossible to commit silently: put a uniqueness constraint on cluster membership — one row per item per membership table. Then if a miscounting scan feeds a resolver that tries to cluster an already-clustered item, the insert fails loudly instead of quietly creating a duplicate grouping, and the whole batch rolls back in its transaction — the same all-or-nothing rollback that can silently lose a batch, here working for you. The guard doesn’t fix the scan, but it converts a silent data mess into an obvious error you fix in the scan — which is exactly where the bug lives. Pair it with a resolver that never finalises an unruled collision (leaves it untouched rather than minting a guess), and even a bad scan can’t corrupt anything; it can only waste your attention.
The short version
A deduplication resolver marks items resolved by clustering or deciding them, but a folded duplicate still matches its twin by raw key — so a scan that recomputes collisions from the raw table re-counts everything already resolved and inflates the backlog. Subtract the resolved set (confirmed-cluster members, decided rows) before grouping, exactly as the resolver does; report off the remainder, not the raw table; and check what’s already resolved before starting a pass. Make sure that resolved set is complete, too: if a second writer records outcomes without updating the ledger your generator reads, the subtraction silently misses them — so every writer updates the shared resume state, or the generator derives “done” from the authoritative outcome. Back it with a uniqueness constraint on membership so a miscount fails loudly instead of corrupting silently.
Frequently Asked Questions
Why does the production resolver get the right number but my scan doesn’t?
Because the resolver drops already-placed items (cluster members, decided rows) before it detects collisions, and your from-scratch scan skips that step. Same keying, different starting set — the resolver subtracts resolved state; the raw scan doesn’t.
My scan does subtract resolved items — why am I still re-emitting finished work?
Then the set you subtract is probably incomplete. If two writers record outcomes but only one updates the resume ledger your generator reads, everything the second writer did is missing from “done” and gets re-emitted — even though it’s authoritative in the system of record. Make every writer update the shared resume state, or derive “done” from the authoritative outcome table instead of a separate ledger. The cheapest detector: take a freshly-processed key and confirm it appears in the exact place the generator looks next.
A folded duplicate is resolved — why does it still group?
Folding usually leaves the record’s raw fields intact and just points it at a canonical. By raw key it’s still identical to its twin, so a naive grouping still sees a collision. Only the exclusion of placed items tells the two apart.
How do I know I’ve hit this, not a genuinely stuck backlog?
The number won’t fall as you resolve, or it’s an implausibly round multiple of the real remainder. Re-run the scan with the resolved-set subtraction; if the count drops sharply, you were re-counting finished work.
What’s the cheapest safeguard if I can’t fix every scan?
A uniqueness constraint on cluster membership plus a resolver that never finalises an unruled collision. Then a miscounting scan can’t do damage — a double-cluster attempt errors out and rolls back, and nothing unreviewed gets a final value.
Does this only apply to deduplication?
No — it’s any staged resolution where “done” items still resemble “to-do” items by whatever signal a scan recomputes: entity resolution, reconciliation, triage queues. If completed work isn’t subtracted before you measure the remainder, you measure it again.
