You shipped a fix to how one field is computed, re-ran the pipeline across everything, and it reported a clean run. Then the consistency check still flags those records as wrong. The fix is in the code, the rebuild “ran” — and the records never changed. Your own optimisation skipped them.
Table of Contents
Why the gate can’t see a code-only change
A materiality gate hashes inputs: the raw captured fields, the upstream values a record is derived from. On a re-run it recomputes that hash, compares to the stored one, and only re-derives when they differ. This is the single most valuable optimisation in a replayable pipeline — it turns an hours-long full rebuild into a seconds-long “only what changed” pass, and it stops you re-deriving millions of untouched rows on every shared edit.
The blind spot is structural, not a bug: the gate watches the inputs, and a code-only fix changes the function, not the inputs. Same inputs → same hash → “unchanged” → skipped. The pipeline is behaving exactly as designed; it just can’t know that the meaning of “derive this field” changed when the inputs didn’t. (The stored hash is the same mechanism you’d use to fingerprint a record and detect edits — which is exactly why it’s blind here: nothing about the inputs was edited.)
You change how a field is derived. No source input changes.
Hashes source inputs. Inputs unchanged → hash matches.
Gate says "not material". Stored value stays stale. Fix never lands.
Byte-stability flags it — benign, clears on a scoped re-derive.
Same inputs → same hash → skipped. The fix is in the code and merged, but the gate never re-derives the rows it was for.
Same inputs, same hash, skipped — the gate can’t see that the function changed.
The tell
Three signals together, and it’s almost always this:
- A rebuild you expected to touch many records reports most of them unchanged / skipped.
- A downstream consistency or byte-stability check then flags those same records as diverging on a field your fix touched.
- The fresh (recomputed) value is the correct one — the one your fix produces — and the stored value is the old output.
That last point is what separates it from a real problem. If the stored value were right and the fresh one wrong, you’d have a regression to chase. Here it’s a one-way lag: the code moved forward, the stored copy didn’t, because the gate held it back.
Benign lag is not contamination
A benign lag is the code moving ahead of the stored copy — not corruption. Read the recomputed value to be sure.
A consistency check will report this as a divergence on a non-volatile field, which looks alarming — the same shape a genuine data-corruption or cross-contamination bug would make. Resist the reflex; it’s the same discipline as telling a real loss from a stale read. Distinguish them by direction and cause:
- Benign lag: the record was logged unchanged by the rebuild, and the recomputed value is the intended one. The fix simply hasn’t reached it because nothing material changed. Harmless; clears with a forced rebuild.
- Contamination: a shared change bled the wrong value in — the stored value is wrong and the recompute is wrong, or the change touched records it had no business touching. That’s a real incident.
Reading a flagged record’s recomputed value tells you which you have in seconds. Don’t mass-force a rebuild or reach for a “rebuild everything, ignore all gates” hammer before you’ve confirmed it’s the benign case — that hammer often discards other carefully-gated state as a side effect.
How to clear it (when you want to)
Materiality is still correct and you should keep it. The point is that a code-only fix is the one case it can’t catch, so plan for that case explicitly:
- Scoped forced rebuild. Re-derive just the affected records or the affected slice, bypassing the gate — a
--force/--all-recordsmode, or bump the stored hash’s input so the gate re-fires. Scope it to what the fix touched; don’t force the whole dataset (that re-pays the full cost the gate exists to avoid). - Or accept and document the lag. If the stale records are inert — not projected, not user-facing, low-value — it can be cheaper to record the benign divergence (why it’s flagged, that the code is the fix, that it clears on the next forced rebuild) than to spend the rebuild now. A documented benign lag is a known state; an unexplained flag is a future false alarm.
Either way, make it a habit: when a change is code-only, ask “did the gate just skip the records I meant to fix?” before you trust a clean re-run.
The short version
A materiality gate hashes source inputs so a re-run skips unchanged records — the optimisation that makes incremental pipelines cheap. A code-only fix changes a field’s derivation without changing inputs, so the gate skips exactly the records the fix was for: the rebuild reports “nothing to do,” the stored values stay stale, and a consistency check flags a lasting but benign lag. Confirm it’s benign (the record was skipped as unchanged, and the recomputed value is the correct one — not contamination), then clear it with a scoped forced rebuild, or accept and document the lag for inert records. Keep the gate; just remember it is blind to code-only changes.
Frequently Asked Questions
Should I remove the materiality gate to avoid this?
No. The gate is what keeps re-runs from re-deriving everything on every edit — removing it trades a rare, documentable lag for a permanent performance tax. Keep the gate; handle the one case it can’t see (code-only fixes) with a scoped forced rebuild.
How do I tell a benign lag from real contamination?
Read the flagged record’s recomputed value. Benign: it was skipped as unchanged and the fresh value is correct (code moved ahead of stored copy). Contamination: the stored value is wrong and so is the recompute, or records changed that shouldn’t have. Same symptom, opposite cause.
Why not just force a full rebuild to be safe?
Because it re-pays the exact cost the gate exists to avoid, and a “force everything” mode often discards other carefully-gated state as collateral. Scope the force to the records your fix touched.
My fix is in the code and merged — isn’t that enough?
Only for records the gate re-derives. Code-only fixes change the function, not the inputs, so the gate sees “unchanged” and skips them; the stored outputs stay old until something forces a re-derivation. Shipped ≠ applied.
How should I document a lag I decide not to clear now?
Record what’s flagged, that the stored value is stale while the code is correct, which records are affected, and that a forced rebuild clears it. That turns a recurring “why is this flagged?” into a known, intentional state — and stops the next person treating it as a fresh bug.
