Field Notes
Posts tagged “Field Notes”.
- Fabric Without Spark
Microsoft Fabric notebooks usually mean Spark, but there's a pure-Python kernel too. I built a full NYC-taxi medallion pipeline on it — Polars, DuckDB, pandas, delta-rs, no Spark — on the smallest F2 capacity, and benchmarked the three engines. The transformations were easy; the boundaries fought back.
- The Console Gets a Face
I put a thin web front-end on the NHTSA evidence system from the last post. A UI is usually where a system like this starts laundering uncertainty into a confident paragraph — so this one is a form, not a chat box, it keeps official records and consumer complaints in separate columns, and it prints the similarity score instead of hiding it.
- Every Answer Cites Its Source
A vehicle-safety evidence system built end to end in Microsoft Fabric over NHTSA data — SQL answers what's deterministic, semantic search only touches what actually needs language, and every result names a source you can go read. Only three of its eight questions need embeddings at all, and a consumer complaint is never dressed as a defect.
- 72 GB of VRAM, No Graphics Card
I used AMD's Variable Graphics Memory to hand a cheap mini-PC's integrated graphics 72 GB of system RAM as 'VRAM,' then ran a 120-billion-parameter model on it — a poor man's Mac for frontier-sized open models. It works. It also runs at reading speed, because the number that governs local LLMs isn't VRAM size, it's memory bandwidth.
- Deploying Fabric Without a Debugger
I gave myself three Fabric workspaces and a deployment pipeline, and the first thing that went wrong went wrong quietly — the notebook deployed fine and pointed at the wrong environment. A field note on why CI/CD ends up small: when it breaks, you have a log and nothing else.
- willitload: What It Refuses to Do
I wrote a fifteen-line pre-flight check, then spent far longer deciding what the real version must never claim. A field note on building a validator you can actually trust — and letting a corpus I designed to break it tell me where I was wrong.
- The Files That Break Your Bulk Load
Point a bulk load at a folder of files and you're trusting the worst file in the set — which usually loads without complaining. A field guide to which files silently corrupt a load, why, and a fifteen-line pre-flight check that names them before the load runs.
- Why I Self-Host
The reasons I give — privacy, cost, resilience — are all true, and all partly a cover story. I set out to build a little local-AI appliance and spent the night fighting NVIDIA drivers and firewall lockouts instead, and enjoyed it. That last part is the actual reason, and it isn't a recommendation.
- Markdown to PDF Is a Swamp
"Convert this Markdown to a PDF" sounds like a solved problem — a one-liner. It is a swamp: no standard pipeline, a dead-but-ubiquitous engine at its center, and a workflow where you only find out the PDF is wrong after you've made it. I built a tool that doesn't drain the swamp, just hands you an honest map.
- Not Every FLAC Is Lossless
A file's extension is a label, not a promise. Self-hosting a music library, I kept running into FLACs that were lossy files in disguise and 'hi-res' tracks upsampled from a CD. The metadata lies; the spectrogram doesn't. So I built a tool to tell them apart.
- They Took the Jack
The headphone jack was the most universal, highest-fidelity, zero-configuration audio interface ever shipped — and it left the phone because most people didn't miss it. That 'most people didn't' is the whole story. It isn't a conspiracy; it's what designing for the average does to the edges.
- I Automated Astrology in Six Languages. I Don't Believe a Word of It.
I built a machine that writes, voices, and publishes horoscopes in six languages every night, and I'm an atheist. The astrology was never the point — it was the perfect payload to stress-test a localization engine. And localization, it turns out, is the part that refuses to become a pipeline stage.
- Your Site Has Two Readers Now
For twenty years you wrote for a human and for Googlebot, and they wanted roughly the same thing. Now a third party reads your page, answers the question itself, and never sends the visitor. I built a scanner to see my own site the way that reader does — and scored a 93 for search engines and a 79 for answer engines.
- A Council Isn't a Debate
One model has only one gear: forward. It answers, extends, agrees — it has no reflex that says 'wait, are we even on the right road?' I built a tool to make several models argue instead, and learned that fanning a question out to a panel and having them argue are completely different things.
- The Backlinks You Draw Yourself
Obsidian and Notion sell 'connected thinking,' but you draw every connection by hand — the graph just displays the links you already typed. I built the tool that finds them for you instead, then spent months perfecting its architecture and never once checked whether it helped me write.
- The Model Has No Memory. That's a Data Problem.
Every model is a stateless brain riding a stateful transcript — close the tab and it forgets the world you built with it. Giving it durable memory isn't a prompting trick; it's an infrastructure problem, and the first thing it teaches you is that facts and stories have to be stored completely differently.
- The Six Months I Had to Reload
A vendor changed one file and nothing broke — the warehouse just ingested subtly wrong numbers for half a year. Recovering it without the business noticing is what taught me what replayability, recoverability, and validation gating actually mean. Wrong data is worse than no data.
- The Happy Path Ends at Day Two
Every data platform demos the same way: point it at a file, watch it load, applaud. That's day one, and day one is a sales artifact. The job is day two — the 3am failure, the 'what exactly ran,' the redeploy of one broken piece. SSIS's catalog got day two right in 2012 in ways Microsoft Fabric still leaves you to build yourself.
- The Cluster Was Idle the Whole Time
A Fabric notebook took three hours to copy fifty thousand files from ADLS Gen2 into a lakehouse. The Spark cluster behind it — dozens of cores — sat almost completely idle while one driver thread made the round trips one at a time. Distributed compute only speeds up the work you actually hand to it.
- A File Failed Once and Could Never Come Back
Dedup by content hash and you close a door you didn't know you were closing. The day your own bug rejects a good file, you learn the ledger only opens one way — and the only recovery the system offers is to lie to it.
- Two Timestamps Beat a Status Column
A status column answers 'what state is this in now?' and destroys the answer to 'what state was it in last Tuesday?' every time you update it. Replace the one mutable cell with a couple of timestamps and the state becomes something you derive — reversibly, and as of any date you like.
- Cron Is a Clock, Not a Scheduler
Cron fires on time and knows nothing else — not whether the last run finished, not whether the inputs arrived, not whether it already failed twice tonight. Most 'scheduling' bugs are really the gap between a clock and a scheduler.
- You Don't Need the Whole Map — Usually
I keep jumping into things I don't fully understand and figuring them out on the way. It works more often than it should — and fails in a specific way that AI is now making both easier and more dangerous to walk into.
- I Sequenced My Family. We Still Don't Have an Answer.
A rare-disease hunt across my family and my sister's family: one of the kids has a developmental disability and seizures, and after years of specialists, no diagnosis. I'm a data engineer, not a geneticist — but I had the same raw material the labs use, so I sequenced the families and built a pipeline to look. Here's what the doing taught me, including the part where the mystery held.
- The Pipeline Behind Sequencing My Family
The companion to the personal essay: the how. On-demand pipeline assembly with an AI, what FASTQ/BAM/VCF actually are, why data quality is the whole game, the free tools a parent can reach, the South Asian reference-data gap, and why we anchored to the data instead of the phenotype.
- Generate Slow, Don't Slow the Generation
To teach a word you slow it down — but slowing a recording smears the exact sounds a learner needs to hear. The fix was to stop slowing audio and start generating it slow. A small experiment, an honest number, and a second lesson about which sounds you're even allowed to stretch.
- You Can Check a Cookie
We baked a batch that refused to set no matter how long it stayed in the oven. The recipes came from two places — bakers with a track record, and an AI. Guess which ones failed. A cookie, it turns out, is a set of ratios you can eat.
- The AI Wrote the Graphics Code. I Still Had to Know Graphics.
I vibe-coded a browser 3D-text generator in a weekend. It ran on the first try and rendered garbage in three specific, invisible-until-you-look ways — flipped holes, striped shading, opaque 'transparent' PNGs. Each fix needed exactly the graphics knowledge the AI didn't apply.
- Weight Isn't Agreement
RAG hands an AI a stack of documents and lets it produce one confident answer — hiding that the sources disagree, that five of them are the same article reposted, that it should have said 'I don't know.' I built an evidence layer around one rule: never give the weight without the fight.
- Keep the Runner Dumb
Most small data pipelines get cluster-scale tooling bolted onto a laptop-scale problem. I built the opposite — a single-file SQL runner that refuses to parse your SQL, holds no state, and won't schedule anything. The whole design is a stack of things it declines to do.
- Checksum, Not Filename
Every ingestion pipeline has to answer one question first — 'have we already loaded this?' Answer it with the filename and you'll drop the corrections and double-load the reruns. Answer it with the content and the ambiguity disappears.
- Building a Local LLM Box That Doesn't Need Babysitting
A field note on assembling a small inference machine, putting Ubuntu Server on it, and getting a stack that runs stable for daily work without constant firefighting.
- The Keys That Never Matched
A migrated pipeline recomputed the exact same hash keys as the old one and ran green for weeks. Every join on those keys returned nothing — same string, same SHA-256, different bytes, because one side was hashing UTF-16 and the other UTF-8.
- The Missing Column That Didn't Crash
A column vanished from an upstream CSV and nothing broke — no error, no failed job, just wrong numbers for a week, with nothing to announce it. Spark had done exactly what I told it to: map the file to my schema by position, not by name. A loud failure would have been a gift.
- How I Learned to Distrust My Own Demo
The most useful skill I built across a decade of projects isn't a technique. It's the reflex to disbelieve an impressive result until something that doesn't care about my feelings confirms it. That's science, worn as an engineering temperament.
- The Bottleneck Was Always the Bus
A decade of projects across games, big data, and deep learning kept teaching me one thing: the compute was never the expensive part. Moving data was. And chasing that lesson through GPUs in games is what quietly walked me into machine learning.
- I've Been Building the Same Control Loop Since Netcode
The architecture I built to hide network latency in a game — optimism at the edge, authority at the core, reconciliation between them — is the exact one I now reach for to keep AI inside boundaries. Same three moves, different decade.
- Everything With a Battery Cooks in a Parked Car
I wanted parking-mode recording without wiring into the starter battery, in a car that bakes every summer and freezes every winter. Those two rules ruled out almost the entire off-the-shelf market and left me building a battery-less 4G dongle — and learning why truckers don't trust fuses.
- Fail Closed, Not Fast
One small default separates systems that survive reality from ones that only demo well: when a dependency dies, does your system keep going without it, or stop? For anything protective, the safe answer is the annoying one.
- Incremental Load Is Not One Thing
"Incremental load" gets used like a single technique. It's a family of at least eight, each with its own failure mode — and the right move is usually the weakest one that still guarantees correctness.
- Your Date Dimension Is Not Static
A date dimension is supposed to be the one table you build once and never touch. Then someone adds an "IsCurrentMonth" column, and last quarter's report starts quietly changing overnight.
- A Data Pipeline Is a Shape of Data, Not a Shape of Code
Every year or so I get the itch to build a tool that generates data pipelines, and every year I hit the same wall. It took me a while to see that the wall is the whole point.
- I Built Cellular Routers Every Way. Then I Bought One.
For years I built 4G/5G routers from SBCs and x86 boards — CM4, PC Engines, Banana Pi, industrial mini-PCs — across a pile of cellular modems and WiFi cards on OpenWrt and Rooter. Here's what actually mattered, and why for anything I depend on I now just buy the vendor box.
- I Taught an LLM to Query Data in English. The Hard Part Was Trust.
A proof-of-concept that let researchers ask questions of sensitive data in plain language. Two things decided whether it was a tool or a liability: a semantic layer that gave the model meaning, and a verification boundary the model was never allowed to cross.
- The Tutorial Is the Easy 5%
I pointed a 200-line Keras 3D-CNN tutorial at real brain MRIs. The demo ran in minutes; making it touch real data took months, three preprocessing pipelines, 25 GPU-hours, and landed at an honest 61% — which taught me more than a 95% would have.
- AI Is Good at the Work You Can Check
The same coding assistant that flies through one of my projects crawls through another. It isn't about difficulty. The dividing line is whether a machine can tell when the output is wrong.
- I Built a Search Engine by Hand
Inverted index, TF-IDF, cosine ranking — before letting myself reach for a library. The two things that stuck weren't the ranking math: a DataFrame is the wrong shape for an index, and my great-looking demo scored 0.01 on the benchmark.
- You Can't Hotfix Atoms
I have a folder of 3D models named like a codebase — v3, v4, v6, and one branch literally named .rej. I versioned physical parts exactly the way I version software, and the print bed taught me where the analogy breaks. The break is the lesson.
- The NVMe Specs That Actually Matter for a Homelab
Ignore the Gen4-versus-Gen5 number on the box. For a Proxmox box, two boring specs decide everything — and the retailer's spec table will happily lie to you about one of them.
- Migrating a ZFS pool to new drives in Proxmox
Moving a pool to bigger disks with zfs send/receive: build a fresh pool, replicate, destroy the old one, and rename — so Proxmox never notices.
- I Have Four Edit Histories for the Same Photo
Non-destructive photo editing keeps the original untouched and logs your edits separately — event sourcing, basically, for pictures. It's a promise about the original file. It says nothing about whether you can still read the log in ten years.
- Passing physical disks to Proxmox VMs
How to give a VM direct access to a physical disk in Proxmox, and when you'd want to do that versus passing through the whole HBA controller.
- LVM Thin Pools in Proxmox
When to use LVM Thin instead of ZFS in Proxmox, how to set it up, and what over-provisioning actually means in practice.
- Creating a ZFS storage pool in Proxmox
How to add a ZFS pool to Proxmox VE and why it's the right storage backend for most homelab VM and container workloads.
- Identifying ZFS disks with /dev/disk/by-id/
Why /dev/sda is the wrong way to reference disks in ZFS, and how to use persistent identifiers to match pool members to physical drives.
- Netcode Is a Distributed System Wearing a Game
I built multiplayer netcode in grad school and only later realized I'd built a distributed system: optimism at the edge to hide latency, one authority at the core to stay consistent, and a reconciliation loop for when they disagree. The same three moves I keep making in data infrastructure.
- I Put Collision Detection on a GPU. The Timer Said It Was a Wash.
For a grad independent study I built a collision system on the CPU, then ported it to CUDA expecting a speedup. It came out a wash — and that non-result taught me more about parallel compute than a clean win would have.