A Data Pipeline Is a State Machine Wearing a DAG
Last updated on
4 min read

A Data Pipeline Is a State Machine Wearing a DAG


An orchestration canvas can look reassuring: ingest, validate, transform, publish. Four green boxes. Everything downstream of a success arrow has run.

Then someone asks whether the file is safe to replay.

The diagram cannot answer. It shows which tasks executed, but not which durable changes those tasks made. The run succeeded; what state is the data in?

I think this is a weakness of how we commonly draw pipelines. A directed acyclic graph (DAG) is a good representation of task dependencies. It is not, by itself, a model of lifecycle state.

One file can outlive several runs

Consider a synthetic file ingestion flow. Its useful states might be:

StateMeaningSafe next action
ReceivedBytes are durably availableValidate
RejectedValidation failed with a recorded reasonCorrect or replace
AcceptedValidation passed and identity was registeredTransform
PublishedConsumers can query its contributionRetain or explicitly reverse
ReversedA prior contribution was backed outReprocess under a new attempt

A file data-state diagram shows received, rejected, accepted, published, and reversed states, with controlled retry transitions.

Notice that failed is not one of the file states. A run can fail without the file being invalid. The compute session may disappear after validation. A warehouse write may succeed before the orchestrator records success. A monitoring task may fail after publication. Treat all three as “file failed” and retries become dangerous.

I prefer separate records for artifact state and attempt state. The artifact has a stable identity, ideally including a content digest. Each attempt has its own ID, start/end times, error, and evidence of completed commits.

The uncomfortable gap between two commits

Suppose a Microsoft Fabric pipeline writes processed data into a Lakehouse, then publishes a curated table into a Warehouse. Imagine both operations are individually reliable but not part of one cross-system transaction.

At 10:01 the Lakehouse write commits. At 10:02 the Warehouse publish commits. At 10:03 the activity reporting success crashes.

The next run sees a failed orchestration attempt. Should it retry both writes?

If it appends blindly, it may duplicate the data. If it assumes nothing committed, it may delete good data. If it assumes everything committed, it may skip an incomplete publication in a different failure scenario.

The correct recovery behavior depends on durable evidence of which steps committed, not on the final color of the DAG node.

The control record can contain an attempt ID, an input identity, an intended publication scope, and independently verifiable commit markers. Retries then ask what already exists and choose the missing action. This is closer to a small saga than to a monolithic database transaction.

Define transitions by invariants

I like to write transition rules in plain language before writing orchestration activities:

  • Received → Accepted: The source bytes are durable, schema checks pass, and the artifact’s identity is registered.
  • Accepted → Published: Every required output for the intended scope is queryable and reconciled; the publication marker is committed.
  • Published → Reversed: The exact prior contribution has been removed or superseded, with a durable reversal record.
  • Retry: Repeating an attempt must not duplicate or silently alter a contribution that is already committed.

These rules can be tested independently of whether the implementation uses a visual pipeline, notebooks, SQL jobs, or application code.

The important word is scope. “Undo this run” is vague when a run processed many files. “Reverse the contribution associated with artifact X and publication version Y” is something a program can reason about. I explored that boundary further in why backout is harder than loading.

The DAG still matters

The state-machine model does not replace orchestration. Dependencies, concurrency limits, timeouts, and schedules still belong in the DAG. A separate state record also costs storage, maintenance, and disciplined transitions. For an ephemeral one-off transformation, this may be needless machinery.

But for a recurring ingestion service where inputs can be corrected, replayed, rejected, and published independently, the state model earns its keep quickly.

My test is simple: Could a new engineer tell, from durable records alone, whether retrying one input will change the published answer? If the only evidence is a green checkmark in yesterday’s run history, the pipeline is missing a piece of its architecture.

Next in “Microsoft Fabric migration and data correctness” When Backout Is Harder Than Loading →