A Data Pipeline Is a Shape of Data, Not a Shape of Code

Every so often I get the same idea. A tool that generates data pipelines — you describe what you want, it emits the PySpark, you skip the boilerplate. I sketch it, I get excited, and then I hit the same wall I hit last time and quietly shelve it.

For a while I thought the wall was just execution — that I hadn’t found the right abstraction yet. I’ve since decided the wall is telling me something true, and it’s worth writing down, because it changes how I think about this whole category of tool.

Why the CRUD trick doesn’t transfer

You can generate a CRUD app. Point a scaffolder at a schema and it spits out models, forms, endpoints, an admin screen. It works because a CRUD app is a shape of code. The structure is the same every time; only the names change. Fill in the blanks and you’re done.

A pipeline feels like it should work the same way. It doesn’t, and here’s the line that finally made it click for me:

A data pipeline isn’t a shape of code. It’s a shape of data. And columns plus types are exactly where naive templating goes to die.

Take a transform that looks trivial to templatize:

df = df.withColumn("age_bucket", bucket_age(col("age")))

Generate that from a template and it breaks the instant reality shows up. The column is Age, not age. Or age arrives as a string because someone upstream quoted it. Or it’s null half the time. Or it isn’t there at all this month because the source changed and nobody said anything. The code is fine. The code was always fine. The data underneath it moved, and the template had no idea, because a template only knows about syntax and the problem was never syntax.

One template, three ways the data breaks it — and not one of them is a code problem.

That’s the difference. CRUD scaffolding can ignore the data because the data conforms to the schema by construction. A pipeline can’t, because handling data that doesn’t conform is the entire job.

The reframe: schema as the contract

Once I stopped fighting that and accepted it, the design flipped in a useful way.

If the hard part is the data’s shape, then the shape has to become a first-class thing the tool knows about — not an assumption buried in the generated code. Lift the schema up into an explicit contract. Then the notebook or the generated script stops being the source of truth and becomes what it should have been all along: a friendly human surface sitting on top of a strict contract that does the actual guarding.

Concretely, the generated code should open with something that fails loudly:

validate_schema(df, expected={
    "age": "int",          # not "Age", not string, not optional
    "customer_id": "string",
})

Missing column, wrong type, unexpected null profile — stop now, at the edge, with a clear message. Not three transforms later with a NullPointerException that means nothing to anyone. The principle is simple: let the deterministic layer own the part that has to be correct, and let the convenient layer be convenient on top of it.

Ask for roles, not column names

The other thing that falls out of this: if you’re going to generate a pipeline with someone, don’t ask them to type column names into a box. Ask them which column plays which role.

Which column is the primary key? Which one is the event time? Which ones are PII? Give them dropdowns populated from the real schema you already parsed, so they’re choosing from what’s actually there instead of free-typing a name that may or may not exist. You get grounded input, and the tool gets semantics it can act on. “This is the event-time column” is worth infinitely more to a generator than a string that happens to say created_at.

Where I always land

Here’s the part I’d rather not admit, and it’s why the thing stays shelved.

The moment you take schema seriously, the tool wants to know about your schemas — your sources, your lookups, your enrichment tables. And those live inside a platform. So the useful version of this isn’t a standalone product you paste code into; it’s something that runs where the data already is. The generator wants to be a plugin, not a website. That’s not a failure of the idea. It’s the idea telling you where it belongs.

I also don’t think it should ever be fully automatic. The fantasy is “point it at everything and it builds the pipeline.” The reality is that the interesting decisions — which source wins a conflict, what a valid row even is, which date matters — are judgment calls, and a generator that makes them silently is a generator that’s confidently wrong. Human-guided, machine-accelerated. Let it do the typing. Keep the deciding.

So I probably won’t build the version I keep sketching. But every time I try, the wall teaches me the same thing, and it’s a thing worth more than the tool would’ve been: a pipeline is not code with the values swapped out. It’s a running argument with your data about whether it is what it claimed to be.