ETL
Posts tagged “ETL”.
- I Tried to Make Structured File Intake Boring
I tested one configuration-driven intake path against clinical-trial, provider-registry, and public-transit files, including bad rows and retries.
- Fabric Without Spark
I ran an NYC taxi medallion pipeline in Fabric's Python notebook on an F2 capacity, using Polars, DuckDB, pandas, and delta-rs without Spark.
- The Limits I Built Into willitload
willitload checks file structure without semantic guesses, distinguishes name- and position-bound loads, and tests itself against adversarial files.
- The Files That Break Your Bulk Load
A field guide to schema drift, inconsistent file sets, malformed CSVs, and a small preflight check that identifies nonconforming files before a bulk load.
- The Cluster Was Idle the Whole Time
A Fabric notebook spent nearly three hours copying 50,000 files serially from its driver while the Spark executors remained idle.
- How Twelve Rejected Files Became Unreloadable
A content-hash ledger recorded rejected files as already seen, leaving no supported way to retry them after the validation bug was fixed.
- Checksum, Not Filename
Filenames cannot distinguish corrected files from reruns. Content fingerprints give ingestion pipelines a stable identity for duplicate detection.
- The Missing Column That Didn't Crash
Spark mapped a shortened CSV to an explicit schema by position, so the job stayed green while values silently shifted into the wrong columns.
- Incremental Load Is Not One Thing
Eight incremental-loading patterns, the source assumptions behind each one, and the timestamp-watermark edge cases that can silently skip rows.
- Why I Keep Shelving My Pipeline Generator
Repeated attempts to generate PySpark pipelines failed at the same boundary: names and types were insufficient without schema contracts and business roles.
- AI Is Good at the Work You Can Check
Coding assistants saved time when tests, schemas, or reconciliations could check their work; without those checks, plausible mistakes consumed the gain.