On a Tuesday in March, twelve files failed. Not because they were bad — because we were. A lookup table was missing a handful of entries, so the validator threw out every row in all twelve, marked the files rejected, and moved on. We found the missing entries and fixed them inside an hour. Then we spent the rest of the day discovering that the twelve files had nowhere to go.

The pipeline had done nothing wrong. It had done exactly what I’d designed it to do, which is the more uncomfortable kind of bug.

The ledger that remembers too well

The first question any ingestion pipeline answers is have we already loaded this? I’ve argued before that you should answer it with the file’s contents, not its name — hash the bytes, not the filename, so a re-sent file and a genuinely-new file can’t be confused. I still believe that. It’s the right call, and it’s the reason a rerun of the whole night can’t silently double-count anything: the ledger has seen these bytes, so it skips them.

The part I hadn’t thought through is that “seen” is not the same as “loaded.” Our landing notebook skipped any hash already in the ledger — a Delta control table keyed on the content hash — and the ledger recorded rejects too, not just loads. The moment the validator threw out those twelve files, their hashes were burned. As far as the pipeline was concerned, it had seen that content and was done with it, forever. Fixing the bug changed nothing. The files sat in a reject folder, correct now, and completely invisible to the system that had rejected them.

There was exactly one way to get them back in, and no runbook described it because no one had planned for the situation: rename the files so their bytes changed, and let them arrive as strangers. Reprocess by lying to the ledger — manufacture a fake “new file” event for a file that was neither new nor, this time, wrong.

That’s when it landed. Content-hash dedup is a one-way door. “Have we seen this?” keyed on what’s inside a file is a permanent claim, and it’s permanent whether the file was good or bad. You build the ledger to protect yourself from re-processing, and the day your own code is wrong is the day you find out the door only opens in one direction.

Building the reverse gear on purpose

The fix wasn’t to weaken the ledger. It was to build the thing that should have shipped alongside it: a deliberate way to reverse a decision. It came down to three verbs, and keeping them separate is most of the work.

• Requeue — make a file eligible to be read again. Move it back to the inbox, clear its terminal state, and this time it isn’t a stranger; it’s the same file, on the record, retried on purpose.
• Replay — the file loaded fine, but a downstream transform was wrong. Don’t touch the file or re-read it. Delete the rows it produced in the layers below and rebuild just those, from data you already trust.
• Backout — you already accepted data that turned out to be wrong. Delete it first, everywhere it landed, then requeue.

Most “just reload it” requests are secretly one of those three, and running the wrong one is how you turn a small fix into a reconciliation nightmare — a requeue where you needed a backout leaves the old rows sitting underneath the new ones.

The rule that saved us from the rest of the footguns: the reason a file was rejected is not the decision about whether to reload it. A file thrown out because our schema check was too strict is recoverable. A file that is genuinely empty is dead — requeue it and it will simply fail again, and now you’ve spent a reload on a file that was never coming back. So the reverse gear’s first job isn’t to reload anything. It’s to sort the rejects into buckets — recoverable, dead-on-arrival, already-loaded-so-back-it-out-first, needs-a-human — and only the first bucket earns a second run. When in doubt it refuses and asks; it fails toward the safe answer, not the convenient one.

Friction is the feature

The obvious way to ship this is a pipeline with a Run button. We deliberately didn’t, and the reasoning is the whole point.

Routine orchestration should be frictionless — it runs every night, no one should have to think about it. This is the opposite of routine. Every use of the reverse gear is hand-editing history: deleting rows you already wrote, moving files back to a folder they’d left, flipping states that were supposed to be final. That is exactly the kind of operation that should be hard to do by accident.

So the tool defaults to a dry run. Point it at the twelve files and it prints, in full, what it would delete and move and touch — and touches nothing. To actually execute, you have to type a phrase that names the blast radius: RELOAD 12 FILES. Not a checkbox, not an “are you sure,” a string you have to look at the count and write out. Fat-finger it and nothing happens. The friction isn’t an oversight I’ll smooth away later. It’s the safety rail, and it’s proportional to the fact that the operation is irreversible in the other direction too.

The part I’d tell my earlier self

The ledger was never the mistake. It’s the reason the pipeline is safe to rerun at all, and I’d build it the same way tomorrow. But “we’ve seen this before” is a statement about the past, and the past is precisely the thing you occasionally have to correct — a bug on your side, a lookup that got filled in an hour too late, a file that was fine the whole time and got caught in someone else’s mistake.

A pipeline that can only ever move forward isn’t robust. It’s pointed one direction and unable to turn around, and it feels fine right up until the first day you need it to. Build the reverse gear the day you build the ledger — not the Tuesday you finally need it and find the door has already closed behind twelve perfectly good files.