The system said the records were gone. They really were gone. But the same system routinely reports counts that are wrong by a factor of eight — so how do you know which kind of “gone” you are looking at, before you act on it?
The system of record is the one authoritative store of a fact; a stale read is any view that can report that fact incorrectly because it is cached, replicated, or eventually consistent.
Table of Contents
The incident, in numbers
The morning run created 29 records in a third-party system. Every one returned success; the local log recorded 29 creations.
By evening they were gone — not “looked gone in one place”, gone. A directory search for a record made that morning returned zero matches, and the system’s own roster showed a count 29 short of where the morning had left it. Two independent reads agreed the work was missing.
So it was recreated: 25 of the 29 went back in, each confirmed present immediately after; the rest were already accounted for. Afterward the roster told the whole story at a glance:
- morning total after creation: 29
- evening roster: the 29 absent
- records recreated: 25
- roster after recovery: 54
Fifty-four is twenty-nine plus twenty-five. The recovered total is fully explained by the original work plus the recovery — which means nothing “came back.” The loss was real, and the recreation was the right call.
The recovered count equalled the loss plus the recreation, exactly. Nothing had returned on its own.
The belief I had to correct
Here is the uncomfortable part, and the reason this incident is worth writing down.
My first recorded conclusion was the opposite: that the data had not been lost, that the system had merely suffered a transient read glitch, and that the records had reappeared on their own. That note carried a clear instruction to a future reader — do not recreate on a missing read, you will make duplicates.
That conclusion was wrong, and the arithmetic above is what disproves it. If the records had reappeared, the evening roster would have climbed back toward 29 on its own and the 25 recreations would have pushed it to something near 54 with duplicates. Instead the numbers were clean: 29 lost, 25 recreated, 54 total, and — checked weeks later against a full crawl — exactly one record per identity, zero duplicates. The records did not reappear. They were recreated, correctly, once.
I let a plausible story (“eventual consistency, it’ll settle”) override the ledger. The ledger was right the whole time.
A plausible explanation that contradicts your own arithmetic is a story, not a diagnosis.
Why both readings were credible
What makes this genuinely hard — not just a lesson in carelessness — is that the “transient glitch” theory was not stupid. The same system’s read layer is unreliable, and I had the evidence to prove it.
On other runs, the roster reported 20 users while the local ledger held 165. It reported 80 one minute and 201 the next, with no writes in between. Those are not real changes; they are a read layer serving stale or partial views. Anyone watching those numbers learns, correctly, that a low read is often a lie.
So the environment trained me to distrust exactly the signal that, this one time, was telling the truth. That is the core difficulty: a system that usually lies about loss can still, sometimes, actually lose your data — and the two cases present identically in a single glance.
The environment trained me to distrust the one signal that, this time, was true.
How to tell them apart — before you act
The resolution is not “trust the system” or “distrust the system.” It is to stop relying on the untrustworthy view for a decision this consequential.
- Keep a local ledger the remote system cannot touch. The one artifact that survived the wipe was the local record of what had been created. It is the anchor every other decision hangs from — without it, “is this loss real?” has no reference point. This is the same instinct as trusting the authoritative store over a cached count: the number you can reconstruct beats the number you are shown.
- Confirm across time and a second view. A single read is an opinion. The loss here was believed only after a search and a roster count agreed, and stayed agreed. When the two disagree, or one settles on a re-check, you are looking at a stale read, not a loss.
- Do the arithmetic, and let it overrule the story. Counts that add up cleanly (loss + recovery = total, no remainder) confirm a real loss; counts that overshoot reveal duplicates from acting too soon. The math is a harder authority than any narrative about consistency.
- Never mass-recreate on one read. The expensive failure mode is recreating on a false alarm and doubling your data. The check that makes recreation safe is a per-item existence test against the system of record — reconcile before you recreate, recreating only the genuinely absent and verifying each after.
What the ledger cannot do
A local ledger tells you what should exist; it cannot, by itself, tell you what does exist in the remote system at this instant — that still requires a read, and the read is the unreliable part. The ledger’s job is narrower and vital: it converts an unanswerable question (“is this loss real?”) into an answerable one (“does the authoritative store match my record, confirmed twice?”). It also cannot prevent the loss; nothing here would have stopped a downstream wipe. What it prevents is the second disaster — the duplicates you create by reacting to the first one badly. Detecting the loss and reacting safely are different problems, and only the ledger-plus-arithmetic discipline solves the second.
A record a check flags as wrong is often the same illusion in another guise — the code moved ahead of a stored copy an incremental gate skipped, not corruption; see The fix that didn’t take.
The short version
Twenty-nine records created in the morning were genuinely gone by evening; a search and a roster count agreed, and recreating 25 of them (the rest already present) was correct — proven because the resulting total, 54, was exactly 29 plus 25, with zero duplicates on a later full crawl. My first conclusion had been the opposite — a transient read glitch, records “reappeared”, do not recreate — and it was wrong, overturned by that arithmetic. The reason both readings were credible is that the system’s read layer genuinely is unreliable (it reported 20 against a real 165, and 80 then 201 with no writes). The discipline that resolves it: keep a local ledger the remote system cannot touch, confirm across time and a second view, let the arithmetic overrule the story, and never mass-recreate on a single read.
Frequently Asked Questions
How do I know if data is really lost or just not showing up?
Do not rely on one read. Confirm the absence across time and through a second, independent view — a search and a count, minutes apart. A real loss stays absent under every check; a stale read disagrees with another view or settles when you look again. And reconcile against a local record of what should exist, so “missing” is measured against an anchor the remote system cannot alter.
Why is recreating missing data dangerous?
Because if the data was not actually lost — only mis-reported — recreating it produces duplicates in the system of record, which is often harder to clean up than the original scare. The safe form is a per-item existence check: recreate only the records genuinely absent from the authoritative store, skip those present, and verify each afterward.
What is a system of record, and why keep a separate ledger?
The system of record is the one authoritative store of a fact. A separate local ledger is your own log of what you did to it — what you created, when. It matters because when the system of record becomes unreadable or loses data, the ledger is the only surviving reference for what should be there, turning an unanswerable “is this real?” into a checkable comparison.
Can a system both lose data and report false counts?
Yes, and that combination is the hardest to reason about. A read layer that is cached, replicated, or eventually consistent can serve wrong counts routinely; that same system can also suffer a genuine loss. Because the two present identically in a single glance, an environment full of false alarms trains you to dismiss the one alarm that is true.
How did arithmetic prove the loss was real?
The recovered total equalled the original count plus the number recreated, with no remainder — 29 plus 25 equalled 54. Had the “records reappeared on their own” theory been true, the recreations would have pushed the total past 54 into duplicate territory. A later full crawl showed exactly one record per identity, confirming the clean sum and refuting the reappearance story.
What should I do the moment a system reports missing data?
Stop before acting. Record the reading, then take a second one through a different view and after a short interval. Compare both against your local ledger. Only if the absence is consistent across views and time, and the ledger agrees the records should exist, do you recreate — and even then item by item with an existence check, never as a bulk re-run on the strength of the first alarm.
