All posts

At 10,000 records SQLite already beat unindexed JSONL on lookup, aggregate and update, while append stayed close. The operations decided it, not the row count.

I tested one configuration-driven intake path against clinical-trial, provider-registry, and public-transit files, including bad rows and retries.

I put a small DuckLake, DuckDB, and marimo data platform through one complete job, from 200,000 to 100 million events.

A paced NVDEC-CUDA-NVENC pipeline found the live-video capacity boundary and showed why a NumPy round trip cost more than the CUDA work.

WireData turns JSON responses and page tables into local SQL datasets, but the real design problem was preserving consent, bytes, and lineage.

A paced I/Q replay found where an eight-core SciPy pipeline began missing deadlines and dropping blocks, and how far an RTX GPU moved that boundary.

Six implementations of one row-scoring operation show why a 324× resident GPU kernel becomes 10.4× once host-device transfers are included.

A complete GPU ML pipeline reached 4.74× at one million rows, while transfer costs and tiny inference batches often kept the advantage on CPU.

GPU spatial joins crossed by one million points, a rectangle barely won at 100 million, and one quadtree setting silently lost matches.

CAGRA searched vectors up to 76× faster than exact cuVS, but its index build required roughly 45,000 queries to amortize.

One late CPU island barely hurt a Spark RAPIDS plan; two separated islands made the hybrid execution 33% slower than CPU.

Across 100,000 to 200 million rows, GPU joins crossed early, strings never crossed, and partitioned Polars handled data larger than VRAM.

Five Parquet readers show cuDF leading large scans, PyArrow keeping two small projected reads, and projection mattering more than engine choice.

I ran an NYC taxi medallion pipeline in Fabric's Python notebook on an F2 capacity, using Polars, DuckDB, pandas, and delta-rs without Spark.

An NHTSA evidence interface limits users to eight validated questions, separates record types, and labels approximate retrieval with similarity scores.

A Microsoft Fabric evidence system routes five NHTSA question types through deterministic lookup and three through cited semantic retrieval.

A mini-PC assigned 72 GB of system RAM to its iGPU and ran a 120B model, but memory bandwidth limited generation to reading speed.

A Fabric notebook kept its development lakehouse binding after deployment. Parameterization and deliberate logging made the release diagnosable.

willitload checks file structure without semantic guesses, distinguishes name- and position-bound loads, and tests itself against adversarial files.

A field guide to schema drift, inconsistent file sets, malformed CSVs, and a small preflight check that identifies nonconforming files before a bulk load.

Privacy, cost, and resilience matter, but the honest reason I self-host is enjoying the work, even when it means drivers and firewall lockouts.

Markdown-to-PDF is always a chain of parsers and renderers. I compared common engines and built separate fast and fidelity previews to expose their differences.

Some FLACs are transcoded lossy files and some hi-res tracks are upsampled CDs. Spectral inspection can test what the metadata merely claims.

The headphone jack was universal and zero-configuration, yet disappeared because most people did not miss it—the cost of designing for the average.

I automated horoscope generation, speech, video assembly, and publishing in six languages. Translation scaled; register, voice, and cultural expectations did not.

Scanning the same pages for SEO and answer-engine visibility exposed meaningful differences in crawler access, parsing, and citation readiness.

I compared parallel model responses with a sequential, stateful debate in which each model could rebut or concede points made in earlier turns.

A writing tool extracted entities and contradictions, but months of data-model refinement happened before testing whether it improved a real manuscript.

A durable memory design stores facts and narrative history outside the model, retrieves them differently, and reviews every proposed update.

A vendor file changed shape without failing. Retained raw inputs later made it possible to rebuild six months of history without taking reporting offline.

SSISDB and Microsoft Fabric show why run history, diagnosis, version comparison, targeted deployment, and rollback matter after the demo.

A Fabric notebook spent nearly three hours copying 50,000 files serially from its driver while the Spark executors remained idle.

A content-hash ledger recorded rejected files as already seen, leaving no supported way to retry them after the validation bug was fixed.

Write-once timestamps can derive current and historical status for one-way lifecycles; repeating transitions still require an event table.

Cron can start a command at a given time. Overlap prevention, input readiness, failure state, and missed-run recovery require additional state.

Learning by moving before the whole map is clear often works, but AI makes the same habit easier to attempt and harder to recognize when it fails.

I sequenced two families and built a rare-disease analysis pipeline. The work produced lessons about evidence and uncertainty, but no diagnosis.

The engineering behind a family exome analysis, from scan assembly and FASTQ/BAM/VCF processing to read validation and reference-population gaps.

Fresh speech synthesis preserved sharper consonants than phase-vocoder stretching in 27 of 30 pairs, but extreme slowdown introduced new artifacts.

An AI cookie recipe never set, while established recipes did. Ingredient ratios provided the deterministic check that plausible prose could not.

An AI-generated three.js app ran immediately but mishandled glyph winding, bevel normals, and post-processing alpha until visual checks exposed each bug.

A proof of concept that scores support independently from source disagreement, discounts duplicated evidence, and preserves unresolved conflicts.

A single-file SQL runner stays reliable by refusing to parse SQL, hold state, or schedule work—the boundaries small pipelines often need.

Filenames cannot distinguish corrected files from reruns. Content fingerprints give ingestion pipelines a stable identity for duplicate detection.

A field note on assembling a small inference machine, putting Ubuntu Server on it, and getting a stack that runs stable for daily work without constant firefighting.

A migrated pipeline hashed identical-looking values but joined nothing because the old system used UTF-16 and the new system used UTF-8.

Spark mapped a shortened CSV to an explicit schema by position, so the job stayed green while values silently shifted into the wrong columns.

A GPU benchmark, an MRI classifier, an imbalanced split, and a text model all produced numbers that needed more scrutiny than celebration.

Examples from game engines, search, distributed processing, deep learning, and ETL where changing data layout or movement mattered more than changing the arithmetic.

Client prediction, server authority, reconciliation, and event logs offer a practical analogy for placing probabilistic AI inside deterministic software.

Heat, cold, and a rule against using the starter battery eliminated most parking-mode recorders and led to a battery-less 4G build.

DNS pinning, liveness probes, and network rules made a VPN-dependent service stop traffic when its tunnel failed instead of silently exposing it.

Eight incremental-loading patterns, the source assumptions behind each one, and the timestamp-watermark edge cases that can silently skip rows.

Calendar attributes are stable, but columns such as IsCurrentMonth depend on query time. Persist the former and compute the latter in the serving layer.

Repeated attempts to generate PySpark pipelines failed at the same boundary: names and types were insufficient without schema contracts and business roles.

Years of building 4G/5G routers from SBCs, x86 boards, and mixed modems showed what mattered—and why dependable links now get a vendor box.

Between enterprise platforms and one-off query tools lies a missing class of focused utilities for reconciliation, schema drift, and file comparison.

A patient-data text-to-SQL prototype needed semantic metadata to draft useful queries and a human review gate before anything could be trusted.

A 3D-CNN tutorial ran in minutes; applying it to real brain MRIs took months and 25 GPU-hours to reach 61% accuracy against a 52% baseline.

Coding assistants saved time when tests, schemas, or reconciliations could check their work; without those checks, plausible mistakes consumed the gain.

Building an inverted index, TF-IDF, and cosine ranking exposed two failures: a dense matrix exhausted memory and mismatched IDs ruined precision.

Parametric 3D design behaves like code generation, but every bad build still costs hours and material because physical mistakes cannot be hotfixed.

For a Proxmox homelab, sustained writes and endurance matter more than Gen4 versus Gen5—and retailer specification tables can be wrong.

Moving a pool to bigger disks with zfs send/receive: build a fresh pool, replicate, destroy the old one, and rename — so Proxmox never notices.

Non-destructive photo editors preserve the original but scatter edit history across proprietary catalogs whose long-term readability is not guaranteed.

How to give a VM direct access to a physical disk in Proxmox, and when you'd want to do that versus passing through the whole HBA controller.

When to use LVM Thin instead of ZFS in Proxmox, how to set it up, and what over-provisioning actually means in practice.

How to add a ZFS pool to Proxmox VE and why it's the right storage backend for most homelab VM and container workloads.

Why /dev/sda is the wrong way to reference disks in ZFS, and how to use persistent identifiers to match pool members to physical drives.

A concise field guide to the three storage stacks every Proxmox homelab eventually has to choose between — and the gotchas that bite after the fact.

Building multiplayer netcode forced me to handle latency, authority, stale state, correction, and ordered events long before I used those terms in data systems.

A CUDA collision detector was no faster at the tested scene size because of memory layout, transfer costs, divergent work, and coarse timing.