The number was six months.

A vendor changed the format of one of the flat files they sent us — a column shifted, without a flicker of warning, in an otherwise routine delivery. Nothing errored. The load ran green that night and every night after. For half a year the warehouse ingested subtly wrong numbers, served them into reports, and looked completely healthy the entire time. Then someone in the business squinted at a figure that couldn’t be right, and I went looking, and found it had been wrong since a Tuesday two quarters earlier.

This was the on-prem Informatica world — mature, structured, tabular data, a stack that had been doing this reliably for a decade. Flat files from outside vendors were the daily reality, and flat files drift: a column appears, a delimiter changes, an encoding swaps, a file just doesn’t show up. I had a whole bestiary of guards for that — workarounds for missing files, for schema changes, for the usual ways a feed goes wrong. This one walked straight past all of them, because the changed file was valid. It parsed. The numbers it produced were plausible. It was wrong in the one way nothing was watching for.

Wrong data is worse than no data

The disaster wasn’t the day I found it. The disaster was the six months before that, when wrong data sat in the warehouse looking exactly like right data, feeding decisions nobody thought to second-guess.

That asymmetry is the whole thing. A pipeline that stops gets noticed and fixed within the hour — an empty report is annoying, but it’s honest; it tells you it’s empty. A pipeline that keeps running on bad input hands you a confident, wrong answer and lets you act on it for two quarters. A loud failure is a gift; an invisible one is a sitting duck you don’t know you’re feeding. Given the choice, I want the thing to fall over and page me, every time, over calmly writing six months of numbers I’ll later have to disown.

Recovering it without the business noticing

Finding the corruption was an afternoon. Fixing it was the week that actually taught me something.

I couldn’t just wipe the affected tables and reload. The business was live on that warehouse — reports ran every morning, people made calls off them. “Take it down for a few days while I reprocess two quarters” was not an available sentence. I had to correct six months of history underneath a system in continuous use, without anyone feeling a bump.

What made that possible was one undramatic fact: I’d kept the raw files. So the recovery wasn’t a reconstruction from the corrupted tables — it was a re-derivation from source. Reprocess the affected window from the original inputs with the corrected logic, build the fixed data alongside the live data, reconcile the two until the numbers agreed where they should and differed only where they should, then swap the corrected slice in — in pieces, oldest to newest, so nothing ever went dark. Boring, careful, high-stakes work, the kind where you triple-check a WHERE clause because it’s touching real history that real people are reading that morning.

That week is where two words I’d only ever used on whiteboards turned into muscle.

The three things it burned in

Replayability. You must be able to re-run any slice of history and get the same answer — from raw, deterministically. If you can’t reprocess last March on demand, cleanly, without a heroic effort, you don’t have a pipeline; you have a one-way street, and you’re one undetected bug away from being stuck on it. Keep the raw inputs. Make reprocessing a routine operation, not an emergency. I build that reverse gear on purpose now, before I need it — requeue, replay, back out — because the version of me improvising it under a live warehouse would have paid anything for a button.

Recoverability without downtime. Correcting the record can’t require taking the business offline, because the day you need to correct it is never a day you get to schedule. The slice-by-slice, build-alongside, swap-in, reconcile pattern is something to have ready, not to invent at 11pm with finance waiting on the morning numbers.

Validation gating — refuse the bad load. All of the above is recovery, and recovery is the expensive end. The cheap fix is upstream: a gate that checks the file is shaped the way you expect and rejects it when it isn’t. Assert the columns are where they should be; refuse the delivery that drifted; fail toward the safe answer, which is the annoying one. A gate like that turns six months of silent corruption into one loud rejection on day one — a page, a held file, a five-minute call with the vendor, instead of a two-quarter cleanup. The whole point is to never need the reload at all.

Experience teaches harder than a book

I could have read all three of those in a data-warehousing textbook and nodded along — yes, keep your raw data; yes, validate inputs; yes, design for recovery. They’d have gone in as facts and stayed inert. What made them load-bearing was six months of wrong numbers and a week spent correcting them by hand under a system I wasn’t allowed to stop.

Most of the data bugs I write about now are, underneath, a newer version of this one — a file that changed, a check that wasn’t there, a number that lied for longer than it should have been able to. I recognize the shape faster than I used to. Six months will do that — an absurd tuition for a single lesson, one I’d give a lot to have skipped, and one I draw on nearly every working day.