Data Quality
Posts tagged “Data Quality”.
- I Tried to Make Structured File Intake Boring
I tested one configuration-driven intake path against clinical-trial, provider-registry, and public-transit files, including bad rows and retries.
- The Files That Break Your Bulk Load
A field guide to schema drift, inconsistent file sets, malformed CSVs, and a small preflight check that identifies nonconforming files before a bulk load.
- The Six Months I Had to Reload
A vendor file changed shape without failing. Retained raw inputs later made it possible to rebuild six months of history without taking reporting offline.
- How Twelve Rejected Files Became Unreloadable
A content-hash ledger recorded rejected files as already seen, leaving no supported way to retry them after the validation bug was fixed.
- The Pipeline Behind Sequencing My Family
The engineering behind a family exome analysis, from scan assembly and FASTQ/BAM/VCF processing to read validation and reference-population gaps.
- Keep the Runner Dumb
A single-file SQL runner stays reliable by refusing to parse SQL, hold state, or schedule work—the boundaries small pipelines often need.
- Checksum, Not Filename
Filenames cannot distinguish corrected files from reruns. Content fingerprints give ingestion pipelines a stable identity for duplicate detection.
- The Keys That Never Matched
A migrated pipeline hashed identical-looking values but joined nothing because the old system used UTF-16 and the new system used UTF-8.
- The Missing Column That Didn't Crash
Spark mapped a shortened CSV to an explicit schema by position, so the job stayed green while values silently shifted into the wrong columns.