Why I Keep Shelving My Pipeline Generator

Every so often I get the same idea. A tool that generates data pipelines — you describe what you want, it emits the PySpark, you skip the boilerplate. I sketch it, I get excited, and then I hit the same wall I hit last time and quietly shelve it.

For a while I assumed I simply had not found the right abstraction. Repeated attempts pointed to a more specific problem: the generator knew the desired code pattern but not enough about the data entering it.

CRUD scaffolding does not transfer cleanly

You can generate a CRUD app. Point a scaffolder at a schema and it spits out models, forms, endpoints, an admin screen. It works because a CRUD app is a shape of code. The structure is the same every time; only the names change. Fill in the blanks and you’re done.

A pipeline initially looks similar, but its behaviour depends on incoming data, not only on a repeatable code structure.

Take a transform that looks trivial to templatize:

df = df.withColumn("age_bucket", bucket_age(col("age")))

Generate that from a template and it breaks the instant reality shows up. The column is Age, not age. Or age arrives as a string because someone upstream quoted it. Or it’s null half the time. Or it isn’t there at all this month because the source changed and nobody said anything. The code is fine. The code was always fine. The data underneath it moved, and the template had no idea, because a template only knows about syntax and the problem was never syntax.

One template, three ways the data breaks it — and not one of them is a code problem.

That’s the difference. CRUD scaffolding can ignore the data because the data conforms to the schema by construction. A pipeline can’t, because handling data that doesn’t conform is the entire job.

The schema has to be an explicit contract

The generator therefore needs the data shape as a first-class input rather than an assumption buried in emitted code. With an explicit schema contract, the notebook or generated script can remain a convenient interface while validation guards the pipeline at its edge.

Concretely, the generated code should open with something that fails loudly:

validate_schema(df, expected={
    "age": "int",          # not "Age", not string, not optional
    "customer_id": "string",
})

Missing column, wrong type, unexpected null profile — stop now, at the edge, with a clear message. Not three transforms later with a NullPointerException that means nothing to anyone. The principle is simple: let the deterministic layer own the part that has to be correct, and let the convenient layer be convenient on top of it.

Business roles are more useful than free-typed names

The other thing that falls out of this: if you’re going to generate a pipeline with someone, don’t ask them to type column names into a box. Ask them which column plays which role.

Which column is the primary key? Which one is the event time? Which ones are PII? Give them dropdowns populated from the real schema you already parsed, so they’re choosing from what’s actually there instead of free-typing a name that may or may not exist. You get grounded input, and the tool gets semantics it can act on. “This is the event-time column” is worth infinitely more to a generator than a string that happens to say created_at.

Why the standalone version stays shelved

The moment you take schema seriously, the tool wants to know about your schemas — your sources, your lookups, your enrichment tables. And those live inside a platform. So the useful version of this isn’t a standalone product you paste code into; it’s something that runs where the data already is. The generator wants to be a plugin, not a website. That’s not a failure of the idea. It’s the idea telling you where it belongs.

I also don’t think it should ever be fully automatic. The fantasy is “point it at everything and it builds the pipeline.” The reality is that the interesting decisions — which source wins a conflict, what a valid row even is, which date matters — are judgment calls, and a generator that makes them silently is a generator that’s confidently wrong. Human-guided, machine-accelerated. Let it do the typing. Keep the deciding.

I probably will not build the standalone version I keep sketching. A useful implementation would live beside the schemas and lookup data, ask people to assign business roles, validate inputs before generating transformations, and leave judgment calls visible. At that point it is a platform extension rather than the small code generator I originally intended.