A while back I wrote up a fifteen-line pre-flight check — point it at a folder of files, hand it an expected schema, and it names the ones that will break your bulk load before the load runs. That post was about the problem: the one file in four thousand that loads without complaining and quietly corrupts a table.

This is about what happened when I took the fifteen lines seriously.

The script works. For the common case it’s genuinely all you need, and I meant that. But the moment you try to make it trustworthy — across every encoding, every delimiter, name-bound loads and position-bound loads, thousands of files, files that lie about their own format — the interesting problem stops being detection and becomes restraint. Almost anyone can write code that flags a difference between two schemas. The hard part, the part that decides whether a data engineer keeps the tool or mutes it after the third false alarm, is knowing what the tool must refuse to say.

So this is a field note about a validator called willitload, told through its refusals — because the refusals are the design.

Refuse to judge values

The first and load-bearing decision: willitload reports structure, never data quality. It will tell you a column is missing, renamed, retyped, or that a row is ragged. It will never tell you a value is invalid, out of range, or wrong.

That sounds like a limitation. It’s the opposite — it’s the thing that makes the tool finishable and the results trustworthy. There’s exactly one test for whether a check belongs in the tool:

Can this be answered from the shape of the data alone, without knowing what the data is supposed to mean?

“Column revenue is missing” — yes, that’s shape. “This email is malformed” — no, that needs a rule about what emails are. The first is in scope; the second is a data-quality product, and data-quality products are unbounded, because there is always one more rule. The instant you accept a single value-validity rule, you’ve told users you’re that tool, and you inherit its infinite surface and its infinite maintenance. So the answer to “can it also check that emails are valid?” is a principled no, not a roadmap item. The bright line is what keeps the whole thing small enough for one person to keep correct — the same instinct as keeping the runner dumb.

Refuse to guess at a rename

Here’s the case that looks like it wants cleverness and punishes it. A file arrives where customer_id has become account_id — same position, same type, one name changed. Obviously a rename. Shouldn’t the tool just say so?

No. To confirm a rename you have to leave structure and inspect the data, arguing that two columns “look alike.” But two different ID columns can look identical, and normal day-to-day variation makes the same column look different. Any content-based rename-detector is therefore a guesser, and a confidently-wrong “these are the same column” is worse than silence — it corrupts the very map the user is trusting.

So willitload reports the rename as two facts — customer_id is missing, account_id is unexpected — and stops. It surfaces the evidence (same position, same type, name differs, flagged as a possible rename) and withholds the verdict. The human, who has the context the tool doesn’t, decides. Expose the alignment; withhold the verdict. Rename identity is the user’s declaration, never the tool’s inference.

This matters more than it looks, because it’s the same discipline as the value-quality line, applied to identity: the tool refuses to attribute meaning it cannot see. A validator’s authority comes entirely from never being confidently wrong, and the only way to never be confidently wrong is to not make claims your inputs can’t support.

Refuse to pretend past the physics limit

Some bulk loads bind columns by name; some bind by position. willitload models both — but the two modes can catch genuinely different things, and honesty about the gap is the whole point.

In name mode, a load binds on the column name, so a name mismatch is fatal — it’s the thing that breaks the load. In position mode, the load binds on ordinal position, so the same name mismatch is only a warning — the load tolerates it, but it hints at a corruption the positional load itself is blind to. The rule is one line, and it’s the tool’s spine: your certainty should mirror what the load actually binds on. Flag as fatal what the load will die on; warn about what it can’t see; never pretend to more certainty than the load itself has. It’s the validation cousin of failing closed, not fast.

And then there’s the case at the bottom that nothing structural can catch, and the tool says so out loud. Two same-typed columns swap places — amount and tax, both decimals, trade positions. Under a headerless positional load, nothing about the file’s structure reveals the swap: the count is right, the type at each position is right, and the data is wrong. You cannot fix this by profiling harder, because the only thing distinguishing “the amount column” from “the tax column” is meaning, and meaning is exactly what structure can’t see. So willitload does two honest things instead of one dishonest one: if names are present it surfaces the disagreement as a warning; if they aren’t, it tells you plainly that this is beyond what a structural check can see. A tool that names its own blind spot is more trustworthy than one that pretends it has none — the same reason it’s worth saying clearly where a check stops.

The part where I let it be wrong on purpose

Here’s the thing I actually want to write about, because almost nobody does: every test I wrote, I wrote knowing what I expected to find. That’s the fatal flaw of testing your own work. The fixtures confirm the behaviors you already built; they can’t surprise you, because you encoded your own expectations into them. A tool that passes only the tests its author imagined is a tool that works exactly as well as its author’s imagination — no better.

So I built a second, adversarial corpus, deliberately — a set of folders with problems seeded from independent logic, and, for the blind portion, an answer key I kept separate so I couldn’t peek. The point wasn’t to confirm the tool worked. The point was to break it, and to find out where my own design was lying to me.

It found four real things.

The most important was a mission-versus-behavior mismatch. I’d classified an integer column arriving as text as a “widening” change — technically defensible, since text can hold anything — so the tool marked it a mild warning and let the file pass. But an int column becoming text is the exact corruption my own war story is about: the ID column that picks up one stray non-numeric value and turns the whole column to text, breaking every downstream join. My tool was waving through the precise horror it existed to catch, and I only saw it because a folder I’d built to be adversarial forced the question. Numeric-to-text is now a breaking error, as it always should have been.

The corpus also caught a file-selection bug (a baseline file sitting inside a scanned folder got profiled as data, inventing a phantom schema), a UTF-16 decode-ordering bug (valid UTF-16 files producing garbled column names because the decode happened after the parse instead of before), and a performance gap against my own target. All four were things my own fixtures were structurally incapable of surfacing, because I’d have had to already suspect them to write the test.

That’s the field note inside the field note: your own tests inherit your own blind spots. The only reliable way to find what you can’t imagine is to build something whose job is to be adversarial to you, and then believe it when it disagrees. The corpus that broke my tool is now committed to the repo — it’s simultaneously the regression suite and the honest record that the tool was, at several points, wrong.

What it is

willitload is a stateless, local-first Python engine — one pip install willitload, point it at a fileset, and get back a golden/broken partition with the exact broken files named and the exact reason for each. It uses DuckDB for the reading and type inference, so it clears thousands of files in seconds; Python only orchestrates. It has a name mode and a position mode, four ways to declare the expected shape (a flat schema, a prior scan, a golden sample file, or a SQL CREATE TABLE), and a non-zero exit code on breaking drift so it drops straight into a CI gate. It’s Apache-2.0, free, and on GitHub.

But the features aren’t the interesting part, and I’d rather you took the other thing. The detection was an afternoon. The discipline — deciding what a validator must never claim, and then building test data specifically to catch myself claiming it anyway — was the actual work. A tool you can trust is mostly a list of things it refuses to do, and an honest account of the times it did them anyway before you noticed.

---

Companion to The Files That Break Your Bulk Load, which has the fifteen-line version you can write yourself. This is what it looks like when it has to be trustworthy.