Duckdb
Posts tagged “Duckdb”.
- Fabric Without Spark
Microsoft Fabric notebooks usually mean Spark, but there's a pure-Python kernel too. I built a full NYC-taxi medallion pipeline on it — Polars, DuckDB, pandas, delta-rs, no Spark — on the smallest F2 capacity, and benchmarked the three engines. The transformations were easy; the boundaries fought back.
- willitload: What It Refuses to Do
I wrote a fifteen-line pre-flight check, then spent far longer deciding what the real version must never claim. A field note on building a validator you can actually trust — and letting a corpus I designed to break it tell me where I was wrong.
- The Files That Break Your Bulk Load
Point a bulk load at a folder of files and you're trusting the worst file in the set — which usually loads without complaining. A field guide to which files silently corrupt a load, why, and a fifteen-line pre-flight check that names them before the load runs.
- Keep the Runner Dumb
Most small data pipelines get cluster-scale tooling bolted onto a laptop-scale problem. I built the opposite — a single-file SQL runner that refuses to parse your SQL, holds no state, and won't schedule anything. The whole design is a stack of things it declines to do.
- The Missing Middle in Data Tooling
Enterprise platforms on one end, a pile of interchangeable SQL boxes on the other. The useful middle is mostly empty — and not because it is hard to build.