data-quality
Posts tagged "data-quality".
- The Files That Break Your Bulk Load
Point a bulk load at a folder of files and you're trusting the worst file in the set — which usually loads without complaining. A field guide to which files silently corrupt a load, why, and a fifteen-line pre-flight check that names them before the load runs.
- The Six Months I Had to Reload
A vendor changed one file and nothing broke — the warehouse just ingested subtly wrong numbers for half a year. Recovering it without the business noticing is what taught me what replayability, recoverability, and validation gating actually mean. Wrong data is worse than no data.
- A File Failed Once and Could Never Come Back
Dedup by content hash and you close a door you didn't know you were closing. The day your own bug rejects a good file, you learn the ledger only opens one way — and the only recovery the system offers is to lie to it.
- The Pipeline Behind Sequencing My Family
The companion to the personal essay: the how. On-demand pipeline assembly with an AI, what FASTQ/BAM/VCF actually are, why data quality is the whole game, the free tools a parent can reach, the South Asian reference-data gap, and why we anchored to the data instead of the phenotype.
- Keep the Runner Dumb
Most small data pipelines get cluster-scale tooling bolted onto a laptop-scale problem. I built the opposite — a single-file SQL runner that refuses to parse your SQL, holds no state, and won't schedule anything. The whole design is a stack of things it declines to do.
- Checksum, Not Filename
Every ingestion pipeline has to answer one question first — 'have we already loaded this?' Answer it with the filename and you'll drop the corrections and double-load the reruns. Answer it with the content and the ambiguity disappears.
- The Keys That Never Matched
A migrated pipeline recomputed the exact same hash keys as the old one and ran green for weeks. Every join on those keys returned nothing — same string, same SHA-256, different bytes, because one side was hashing UTF-16 and the other UTF-8.
- The Missing Column That Didn't Crash
A column vanished from an upstream CSV and nothing broke — no error, no failed job, just wrong numbers for a week, with nothing to announce it. Spark had done exactly what I told it to: map the file to my schema by position, not by name. A loud failure would have been a gift.