Data engineering
Microsoft Fabric, Azure data platforms, lakehouse design, ETL migrations, data modeling, incremental loads, watermarks, auditability, and the boring control layers that make pipelines reliable.
- I Tried to Make Structured File Intake Boring
I tested one configuration-driven intake path against clinical-trial, provider-registry, and public-transit files, including bad rows and retries.
- DuckLake on One Machine
I put a small DuckLake, DuckDB, and marimo data platform through one complete job, from 200,000 to 100 million events.
- The Data Was Already in the Browser
WireData turns JSON responses and page tables into local SQL datasets, but the real design problem was preserving consent, bytes, and lineage.
- How Much of a Spark Plan Actually Runs on the GPU?
One late CPU island barely hurt a Spark RAPIDS plan; two separated islands made the hybrid execution 33% slower than CPU.
- DuckDB, Polars, and cuDF on One Analytical Pipeline
Across 100,000 to 200 million rows, GPU joins crossed early, strings never crossed, and partitioned Polars handled data larger than VRAM.
- When Does GPU Parquet Actually Pay Off?
Five Parquet readers show cuDF leading large scans, PyArrow keeping two small projected reads, and projection mattering more than engine choice.
- Fabric Without Spark
I ran an NYC taxi medallion pipeline in Fabric's Python notebook on an F2 capacity, using Polars, DuckDB, pandas, and delta-rs without Spark.
- Every Answer Cites Its Source
A Microsoft Fabric evidence system routes five NHTSA question types through deterministic lookup and three through cited semantic retrieval.
- Deploying Fabric Without a Debugger
A Fabric notebook kept its development lakehouse binding after deployment. Parameterization and deliberate logging made the release diagnosable.
- The Limits I Built Into willitload
willitload checks file structure without semantic guesses, distinguishes name- and position-bound loads, and tests itself against adversarial files.
- The Files That Break Your Bulk Load
A field guide to schema drift, inconsistent file sets, malformed CSVs, and a small preflight check that identifies nonconforming files before a bulk load.
- Building Durable Memory Around a Stateless Model
A durable memory design stores facts and narrative history outside the model, retrieves them differently, and reviews every proposed update.
- The Six Months I Had to Reload
A vendor file changed shape without failing. Retained raw inputs later made it possible to rebuild six months of history without taking reporting offline.
- The Happy Path Ends at Day Two
SSISDB and Microsoft Fabric show why run history, diagnosis, version comparison, targeted deployment, and rollback matter after the demo.
- The Cluster Was Idle the Whole Time
A Fabric notebook spent nearly three hours copying 50,000 files serially from its driver while the Spark executors remained idle.
- How Twelve Rejected Files Became Unreloadable
A content-hash ledger recorded rejected files as already seen, leaving no supported way to retry them after the validation bug was fixed.
- Timestamps Beat a Mutable Status Column
Write-once timestamps can derive current and historical status for one-way lifecycles; repeating transitions still require an event table.
- Cron Is a Clock, Not a Scheduler
Cron can start a command at a given time. Overlap prevention, input readiness, failure state, and missed-run recovery require additional state.
- You Can Check a Cookie
An AI cookie recipe never set, while established recipes did. Ingredient ratios provided the deterministic check that plausible prose could not.
- Keep the Runner Dumb
A single-file SQL runner stays reliable by refusing to parse SQL, hold state, or schedule work—the boundaries small pipelines often need.
- Checksum, Not Filename
Filenames cannot distinguish corrected files from reruns. Content fingerprints give ingestion pipelines a stable identity for duplicate detection.
- The Keys That Never Matched
A migrated pipeline hashed identical-looking values but joined nothing because the old system used UTF-16 and the new system used UTF-8.
- The Missing Column That Didn't Crash
Spark mapped a shortened CSV to an explicit schema by position, so the job stayed green while values silently shifted into the wrong columns.
- The Projects Where Data Movement Cost More Than Compute
Examples from game engines, search, distributed processing, deep learning, and ETL where changing data layout or movement mattered more than changing the arithmetic.
- Incremental Load Is Not One Thing
Eight incremental-loading patterns, the source assumptions behind each one, and the timestamp-watermark edge cases that can silently skip rows.
- Your Date Dimension Is Not Static
Calendar attributes are stable, but columns such as IsCurrentMonth depend on query time. Persist the former and compute the latter in the serving layer.
- Why I Keep Shelving My Pipeline Generator
Repeated attempts to generate PySpark pipelines failed at the same boundary: names and types were insufficient without schema contracts and business roles.
- The Missing Middle in Data Tooling
Between enterprise platforms and one-off query tools lies a missing class of focused utilities for reconciliation, schema drift, and file comparison.