Every bulk job gets re-run — after a crash, a false failure, or a nervous operator. Whether that re-run heals the data or doubles it comes down to one question you answer before it starts.
Reconciliation is the step where a job compares what it is about to create against what already exists, so it acts only on the true difference.
Table of Contents
Why re-runs are the normal case, not the edge case
A bulk job of any size will be interrupted. A connection drops, a machine sleeps, a batch is stopped by hand, half the items fail a transient check. Every one of those ends with the job partly done and the obvious next move being “run it again.”
So the interesting question is never “does the job work?” It is “what does the second run do to the items the first run already completed?” A job that has no answer to that is not finished — it is a duplicate generator waiting for its first retry.
A bulk job with no reconciliation step is not idempotent; it is a duplicate generator on a delay.
The deduplication key: identity, not position
Reconciliation needs to answer “have I already created this?” — which requires a way to say two records are the same real entity. That is the deduplication key, and choosing it well is most of the work.
A good key is stable (it does not change between runs), unique to the entity, and present in both your source data and the system of record so you can match across them. A natural identifier the entity already owns is ideal. A row’s position in a file is the classic bad key: insert one line and every position shifts, so the same entity now looks new.
The subtle failure is a key that is usually unique. If your key is an attribute that two distinct entities can share, reconciliation will treat them as one and skip a record that should have been created — a false skip, quieter and longer-lived than a duplicate. Choose a key that is unique by construction, not by luck.
The local ledger: do not depend on a system you cannot always read
Reconciling against the system of record assumes you can read the system of record. Sometimes you cannot — it is slow, rate-limited, briefly inconsistent, or its list view is paginated in a way that hides most of it.
A local ledger — your own append-only record of what you have created, keyed the same way — is the hedge. It lets a re-run skip everything you know you made without a fragile full scan of the remote system, and it survives even a remote outage. It is not a replacement for the authoritative read; it is a fast, reliable first pass that shrinks how much you must verify remotely. When the two disagree, the system of record wins — but the ledger is what makes the common case cheap and a genuine loss something you can actually measure against a known baseline.
The system of record is the authority; the local ledger is what keeps you from having to trust it blindly on every pass.
Ghost records: present remotely, missing locally
The gap between the two stores has a name worth knowing. A ghost is a record that exists in the system of record but not in your ledger — created, but not written down as created.
They appear for ordinary reasons: a create that succeeded while your process died before it could log the success, a record someone else made, a run whose result you misread. The danger is that a ledger-only reconciliation will not see them, call them missing, and recreate them — a duplicate, from the very step meant to prevent duplicates.
The defense is to let the system of record settle ties. Before creating, check both: if the ledger has it, skip; if the ledger does not but a keyed read of the store finds it, record the ghost in the ledger and skip. Only create when both agree it is absent. A periodic full reconciliation — crawl the store, adopt every keyed record the ledger is missing — keeps ghosts from accumulating.
The fix, assembled
Put the pieces in order and a re-run becomes boring, which is the goal:
- Load the ledger. Everything you already know you created, by key.
- For each source item, resolve its key. Stable, unique, present on both sides.
- Skip what the ledger knows. The cheap, reliable majority.
- For the rest, read the system of record by key. Present → it is a ghost; record it and skip. Absent → create.
- Create idempotently, then read back. Confirm the create landed by reading it from the store — the verdict comes from the record, not the interface message — and append to the ledger.
The result recreates exactly the true difference — no duplicates from ghosts, no false skips from a weak key, no dependence on a remote scan that might be lying.
What reconciliation cannot do
Reconciliation keeps a re-run from duplicating data; it does not keep the first run from losing it, and it does not repair records that exist but are wrong. It answers “does this entity exist?” — not “is it correct?” An update path still has to compare fields, because a present record can hold stale values. And a key that is subtly non-unique will defeat the whole scheme silently, so the key choice deserves more scrutiny than the plumbing around it. Reconciliation makes retries safe; it does not make the underlying data true.
Re-creating what already exists has a measurement cousin: a scan that re-counts already-resolved items and inflates the backlog — Your dedup scan is over-counting.
The short version
Every bulk job is re-run eventually, so the design that matters is what the second run does to already-completed items. Make it reconcile first: pick a stable, genuinely-unique deduplication key present on both sides; keep a local ledger of what you created as a fast, outage-proof first pass; and create idempotently so an attempted duplicate is a skip. Watch for ghosts — records present in the system of record but missing from the ledger — and let a keyed read of the store settle ties before you create, so the anti-duplicate step does not itself make duplicates. Reconciliation makes retries safe; it does not make the data correct, so update paths must still compare fields.
Frequently Asked Questions
What does it mean to reconcile before recreating?
It means that before a bulk job creates anything, it compares its intended records against what already exists — in a local ledger and, for ties, in the system of record — and acts only on the genuine difference. Reconciliation turns “create all of these” into “create the ones that are actually missing,” which is what makes a re-run safe.
How do I choose a good deduplication key?
Pick something stable across runs, unique to the entity by construction, and present in both your source data and the system of record. A natural identifier the entity already owns is best. Avoid positional keys (row numbers shift) and avoid attributes that two distinct entities can share, which cause silent false skips that are harder to notice than duplicates.
What is a ghost record?
A ghost is a record that exists in the system of record but not in your local ledger — created, but never written down as created. It happens when a process dies after a successful write, when someone else creates the record, or when a result is misread. A ledger-only check will call a ghost missing and recreate it, so a keyed read of the store must settle the tie.
Why keep a local ledger if the system of record is authoritative?
Because you cannot always read the authoritative store cheaply or reliably — it may be slow, rate-limited, briefly inconsistent, or paginated so it hides most of its contents. A local ledger is a fast, outage-proof first pass that skips what you know you created, shrinking how much you must verify remotely. The system of record still wins ties; the ledger just makes the common case cheap.
How do I make creation idempotent?
Design it so that creating a record which already exists is a no-op or a skip rather than a second copy — often by having the store reject duplicates on the deduplication key, or by checking existence immediately before the write. Combined with reconciliation, idempotent creation means a mistaken re-run costs nothing, because the operation simply declines to duplicate.
Does reconciliation fix incorrect data?
No. Reconciliation answers whether an entity exists, not whether it is correct. A record can be present and hold the wrong values, so an update path must compare fields rather than just existence. Reconciliation makes retries safe against duplication; keeping the data accurate is a separate discipline built on read-back and field comparison.
