etl
Posts tagged "etl".
- willitload: What It Refuses to Do
I wrote a fifteen-line pre-flight check, then spent far longer deciding what the real version must never claim. A field note on building a validator you can actually trust — and letting a corpus I designed to break it tell me where I was wrong.
- The Files That Break Your Bulk Load
Point a bulk load at a folder of files and you're trusting the worst file in the set — which usually loads without complaining. A field guide to which files silently corrupt a load, why, and a fifteen-line pre-flight check that names them before the load runs.
- The Cluster Was Idle the Whole Time
A Fabric notebook took three hours to copy fifty thousand files from ADLS Gen2 into a lakehouse. The Spark cluster behind it — dozens of cores — sat almost completely idle while one driver thread made the round trips one at a time. Distributed compute only speeds up the work you actually hand to it.
- A File Failed Once and Could Never Come Back
Dedup by content hash and you close a door you didn't know you were closing. The day your own bug rejects a good file, you learn the ledger only opens one way — and the only recovery the system offers is to lie to it.
- Checksum, Not Filename
Every ingestion pipeline has to answer one question first — 'have we already loaded this?' Answer it with the filename and you'll drop the corrections and double-load the reruns. Answer it with the content and the ambiguity disappears.
- The Missing Column That Didn't Crash
A column vanished from an upstream CSV and nothing broke — no error, no failed job, just wrong numbers for a week, with nothing to announce it. Spark had done exactly what I told it to: map the file to my schema by position, not by name. A loud failure would have been a gift.
- Incremental Load Is Not One Thing
"Incremental load" gets used like a single technique. It's a family of at least eight, each with its own failure mode — and the right move is usually the weakest one that still guarantees correctness.
- A Data Pipeline Is a Shape of Data, Not a Shape of Code
Every year or so I get the itch to build a tool that generates data pipelines, and every year I hit the same wall. It took me a while to see that the wall is the whole point.
- AI Is Good at the Work You Can Check
The same coding assistant that flies through one of my projects crawls through another. It isn't about difficulty. The dividing line is whether a machine can tell when the output is wrong.