Data engineering
Microsoft Fabric, Azure data platforms, lakehouse design, ETL migrations, data modeling, incremental loads, watermarks, auditability, and the boring control layers that make pipelines reliable.
- Deploying Fabric Without a Debugger
I gave myself three Fabric workspaces and a deployment pipeline, and the first thing that went wrong went wrong quietly — the notebook deployed fine and pointed at the wrong environment. A field note on why CI/CD ends up small: when it breaks, you have a log and nothing else.
- willitload: What It Refuses to Do
I wrote a fifteen-line pre-flight check, then spent far longer deciding what the real version must never claim. A field note on building a validator you can actually trust — and letting a corpus I designed to break it tell me where I was wrong.
- The Files That Break Your Bulk Load
Point a bulk load at a folder of files and you're trusting the worst file in the set — which usually loads without complaining. A field guide to which files silently corrupt a load, why, and a fifteen-line pre-flight check that names them before the load runs.
- The Model Has No Memory. That's a Data Problem.
Every model is a stateless brain riding a stateful transcript — close the tab and it forgets the world you built with it. Giving it durable memory isn't a prompting trick; it's an infrastructure problem, and the first thing it teaches you is that facts and stories have to be stored completely differently.
- The Six Months I Had to Reload
A vendor changed one file and nothing broke — the warehouse just ingested subtly wrong numbers for half a year. Recovering it without the business noticing is what taught me what replayability, recoverability, and validation gating actually mean. Wrong data is worse than no data.
- The Happy Path Ends at Day Two
Every data platform demos the same way: point it at a file, watch it load, applaud. That's day one, and day one is a sales artifact. The job is day two — the 3am failure, the 'what exactly ran,' the redeploy of one broken piece. SSIS's catalog got day two right in 2012 in ways Microsoft Fabric still leaves you to build yourself.
- The Cluster Was Idle the Whole Time
A Fabric notebook took three hours to copy fifty thousand files from ADLS Gen2 into a lakehouse. The Spark cluster behind it — dozens of cores — sat almost completely idle while one driver thread made the round trips one at a time. Distributed compute only speeds up the work you actually hand to it.
- A File Failed Once and Could Never Come Back
Dedup by content hash and you close a door you didn't know you were closing. The day your own bug rejects a good file, you learn the ledger only opens one way — and the only recovery the system offers is to lie to it.
- Two Timestamps Beat a Status Column
A status column answers 'what state is this in now?' and destroys the answer to 'what state was it in last Tuesday?' every time you update it. Replace the one mutable cell with a couple of timestamps and the state becomes something you derive — reversibly, and as of any date you like.
- Cron Is a Clock, Not a Scheduler
Cron fires on time and knows nothing else — not whether the last run finished, not whether the inputs arrived, not whether it already failed twice tonight. Most 'scheduling' bugs are really the gap between a clock and a scheduler.
- You Can Check a Cookie
We baked a batch that refused to set no matter how long it stayed in the oven. The recipes came from two places — bakers with a track record, and an AI. Guess which ones failed. A cookie, it turns out, is a set of ratios you can eat.
- Keep the Runner Dumb
Most small data pipelines get cluster-scale tooling bolted onto a laptop-scale problem. I built the opposite — a single-file SQL runner that refuses to parse your SQL, holds no state, and won't schedule anything. The whole design is a stack of things it declines to do.
- Checksum, Not Filename
Every ingestion pipeline has to answer one question first — 'have we already loaded this?' Answer it with the filename and you'll drop the corrections and double-load the reruns. Answer it with the content and the ambiguity disappears.
- The Keys That Never Matched
A migrated pipeline recomputed the exact same hash keys as the old one and ran green for weeks. Every join on those keys returned nothing — same string, same SHA-256, different bytes, because one side was hashing UTF-16 and the other UTF-8.
- The Missing Column That Didn't Crash
A column vanished from an upstream CSV and nothing broke — no error, no failed job, just wrong numbers for a week, with nothing to announce it. Spark had done exactly what I told it to: map the file to my schema by position, not by name. A loud failure would have been a gift.
- The Bottleneck Was Always the Bus
A decade of projects across games, big data, and deep learning kept teaching me one thing: the compute was never the expensive part. Moving data was. And chasing that lesson through GPUs in games is what quietly walked me into machine learning.
- Incremental Load Is Not One Thing
"Incremental load" gets used like a single technique. It's a family of at least eight, each with its own failure mode — and the right move is usually the weakest one that still guarantees correctness.
- Your Date Dimension Is Not Static
A date dimension is supposed to be the one table you build once and never touch. Then someone adds an "IsCurrentMonth" column, and last quarter's report starts quietly changing overnight.
- A Data Pipeline Is a Shape of Data, Not a Shape of Code
Every year or so I get the itch to build a tool that generates data pipelines, and every year I hit the same wall. It took me a while to see that the wall is the whole point.
- The Missing Middle in Data Tooling
Enterprise platforms on one end, a pile of interchangeable SQL boxes on the other. The useful middle is mostly empty — and not because it is hard to build.