You imported the record correctly. Then someone fixed a typo in the source — and your re-run skipped right past it, because dedup saw a record that already existed and moved on.
A fingerprint is a compact hash of the specific field values copied into a target, stored at copy time so a later value can be recomputed and compared to detect change.
Table of Contents
Why dedup alone lets edits slip
Deduplication answers one question: have I already imported this identity? Keyed on something stable — a mobile number, an account id — it correctly prevents a second copy of the same entity. That is its whole job, and it does it well.
But “already imported” and “still matches the source” are different questions. Once a record is in the target, dedup will skip it on every future run regardless of what happened to the source in between. So when a person goes back and corrects their own submission — fixes a misspelled name, updates an email — the corrected source record has the same identity as before. Dedup sees the identity, concludes “already done,” and skips. The correction sits in the source forever, never propagating, and nothing complains.
Dedup protects you from copying a record twice. It offers no protection at all from copying it once and then never again.
The fingerprint: hash what you copied, when you copied it
The fix is to record, at import time, not just that you imported a record but what you imported. A fingerprint is the compact way to do that: concatenate the field values you actually wrote to the target, in a fixed order, normalize them, and hash the result to a short string. Store that string alongside the identity.
Two design choices make it reliable:
- Fingerprint only the fields you copy. If a source field never reaches the target, leave it out — a change to it is not drift you can act on, and including it produces alarms you cannot resolve.
- Normalize before hashing. Trim whitespace, fold case, collapse internal spacing, so that cosmetic noise does not read as a change. Otherwise a trailing space someone added produces a “drift” that is not one.
The stored fingerprint is a claim: this is the exact content I put into the target for this identity. Everything else builds on comparing that claim to reality.
Detecting drift: recompute and compare
On a later pass, for each source record, recompute the fingerprint from the current source values using the same fields and normalization. Then compare to the stored fingerprint:
- Equal — the source is unchanged since import; nothing to do.
- Different — the source was edited after import; the target is stale and this record needs re-applying.
- No stored fingerprint — the record predates fingerprinting, so you cannot compare; report it as unknown rather than guessing.
This is a cheap, mechanical check — a string comparison per record — that replaces the impossible task of eyeballing two systems for differences. It reports exactly which records drifted, so a human can re-apply just those, the same way a good pipeline reconciles against the system of record before it acts rather than trusting the copy is still current.
Store what you wrote, recompute what the source says now, and the diff falls out of a single equality check.
What a fingerprint cannot tell you
A fingerprint detects that something in the copied fields changed; it does not tell you which field or what the change was — for that you compare the values themselves, using the fingerprint only as the cheap trigger for a closer look. It cannot detect a change to a field you did not fingerprint, by design. It cannot tell an authorized correction from a mistake — it flags the drift and leaves the judgment to a person. And it is blind to the one edit that also changes identity: if someone alters the very field you dedupe on, the record no longer matches by identity at all, so it reads as new rather than drifted, and a fingerprint keyed to the old identity never gets compared. That case needs a different guard — identity stability, or matching on a second key. The fingerprint is a drift detector for stable-identity records, not a universal change-data-capture system.
Applying it: the audit pass
In practice this becomes a report you can run any time: walk the source, recompute each fingerprint, and print the records that drifted, the ones unchanged, and the ones with no baseline. The drifted list is the actionable output — the records whose corrections have not reached the target and need re-applying, deliberately, by whoever owns that decision.
Because it only reads and compares, the audit is safe to run repeatedly and costs almost nothing. It converts a governance worry — “are the corrections people made actually taking effect?” — into a number you can watch, and turns silent staleness into a visible, short list.
The same fingerprint that detects edits is what an incremental build gate trusts — which is exactly why it is blind to a code-only fix that changes nothing it hashes: The fix that didn’t take.
The short version
Copying records from a source you do not control into a target, then re-running, dedupes by identity and skips anyone already imported — correct for duplicates, wrong for a record whose source was edited after import, whose fix then never propagates and looks done. Store a fingerprint at import time: a normalized hash of exactly the fields you copied, kept with the identity. On a later pass, recompute from the current source and compare — equal means unchanged, different means drifted and stale, absent means no baseline. It is a per-record string comparison that replaces eyeballing two systems, and it feeds an audit that lists precisely which corrections have not reached the target. It cannot see a change to an un-fingerprinted field, judge whether a change was intended, or catch an edit to the identity field itself — that last case needs identity stability instead.
Frequently Asked Questions
Why does dedup skip a record that was edited at the source?
Because dedup only asks whether an identity has already been imported, not whether its content still matches. A person correcting their own submission keeps the same identity — same mobile or account id — so dedup sees “already imported” and skips, regardless of the edit. The correction stays in the source and never reaches the target, and nothing flags it because, to dedup, the record is done.
What exactly should go into the fingerprint?
Only the fields you actually copy into the target, in a fixed order, normalized to remove cosmetic noise — trimmed, case-folded, internal spacing collapsed. Fields you never propagate should be excluded, because a change to them is not drift you can act on and would produce alarms you cannot resolve. The fingerprint represents the exact content you wrote, so a later recompute can tell whether the source still matches it.
How is fingerprinting different from just comparing the records?
It is a cheap trigger rather than the comparison itself. A full field-by-field diff of two systems is expensive and awkward to run often; a fingerprint reduces each record to one short string, so detecting drift is a single equality check per record. When a fingerprint differs, you then compare the actual values to see what changed — but you only pay that cost for the records the fingerprint already flagged.
What if the field I dedupe on is the one that changed?
Then the fingerprint will not help, because the record no longer matches by identity — it reads as a brand-new record rather than a drifted one, and the fingerprint tied to the old identity is never compared. Detecting that requires a different guard: keeping the identity field stable, or matching on a second independent key so a change to the primary identity can still be linked to the original record.
Does a fingerprint tell me what changed?
No — it tells you only that something in the copied fields changed. It is a detector, not a diff. Once it flags a record, you compare the stored and current values directly to see which field drifted and how. This split is deliberate: the fingerprint makes scanning every record cheap, and the detailed comparison runs only on the few that actually changed.
How often should I run the drift audit?
As often as corrections realistically happen and matter — the audit only reads and compares, so it is safe and cheap to run repeatedly. Running it before each import cycle, or on a schedule, keeps the window between a source correction and its detection short. The output is a short actionable list of drifted records, so frequent runs cost little and prevent corrections from sitting undetected.
