When Backout Is Harder Than Loading
Loading a file has a reassuring direction. Read it, validate it, write rows, update the catalogue, move the file to an archive. Every step feels like progress.
In Microsoft Fabric, those steps can cross notebook, Lakehouse, and Warehouse boundaries. Undoing that load is less reassuring. Which rows belong to the file? What if the rows reached the warehouse but the archive move failed? What if the last backout attempt removed half the state and then timed out? And what does retry mean when the retry is allowed to delete things?
I eventually stopped treating backout as the opposite of ingestion. It’s a separate workflow with its own failure states.
There is no single undo button
Suppose a batch crosses three durable boundaries:
- A lakehouse Raw table receives rows.
- A warehouse fact is rebuilt or updated from those rows.
- A file catalogue records the delivery state and a physical file moves to an archive.
The lakehouse transaction doesn’t atomically commit the warehouse write and object-storage move. The system can be correct after each individual operation and still be inconsistent as a whole when execution stops between them.
Each completed step is durable on its own. The workflow needs evidence about which steps finished.
A naive backout script might delete Raw rows, then warehouse rows, then mark the file reversed. If the warehouse delete fails, the catalogue still says loaded, but Raw has already gone. A blind retry may not know whether it should resume, compensate, or stop.
The first design decision is therefore order. Before deleting anything, resolve the exact file identity and affected target scope. Verify that the operation is allowed from the current state. Reject ambiguous cases: two active physical copies, a missing target key, or a competing process already handling the same delivery.
Safe retries need ownership
I like an operation ID for the backout attempt, separate from the original load ID. It lets the next execution ask whether it is continuing the same operation or starting another one.
The state transitions should be conditional. “Mark reversed” is only safe if the catalogue still represents the file and attempt that were checked during preflight. Otherwise a concurrent ingestion or manual correction could change the meaning of the row underneath us.
The durable evidence might be a compact operation journal:
| Step | Evidence before continuing |
|---|---|
| Resolve scope | Exact delivery and target identities |
| Reverse serving data | Warehouse effect is absent or explicitly accounted for |
| Reverse Raw data | No rows remain in the selected source scope |
| Move or quarantine source | Exactly one expected physical location is confirmed |
| Finalize state | Catalogue transition belongs to this operation |
This doesn’t make the workflow atomic. It makes it resumable. A retry can verify completed effects and continue from the next necessary action instead of replaying every deletion blindly.
The details depend on whether downstream facts retain file-level lineage. If a fact aggregates rows from many deliveries, deleting facts by file ID may not even be meaningful. The correct reversal may be a bounded rebuild from the remaining Raw data rather than a row-for-row inverse.
Backout, requeue, and replay aren’t synonyms
I keep these verbs distinct now:
- Backout: remove a previously accepted delivery’s effects from specified targets.
- Requeue: make a delivery eligible for another ingestion attempt.
- Replay: reconstruct downstream outputs from retained source data.
Requeue doesn’t necessarily reload anything immediately. Replay doesn’t necessarily reverse the original file. And a backout should not casually erase the evidence needed to explain the original load.
I learned a smaller version of that distinction when rejected files became unreloadable. A rejected input needs a recovery path. A successfully loaded input that must be unwound needs a much more cautious one.
The strongest argument against elaborate backout machinery is a simpler architecture: if you can retain immutable Raw and rebuild an affected serving window, do that. It often reduces the number of precise inverse operations required. But the original delivery and its operational state still need a coherent fate.
Loading is allowed to be optimistic because a failure usually leaves work unfinished. Backout touches work people may already depend on. That’s why it deserves the more suspicious code path.
Next in “Microsoft Fabric migration and data correctness” Microsoft Fabric Without Spark →