The worst data bug I’ve hit in a while didn’t announce itself. No red pipeline, no stack trace, no failed job. For about a week, every run reported success. The numbers were just quietly, confidently wrong.

The setup is ordinary. A folder fills up every day with small CSV files — call them orders, three columns: order_id, region, amount. A Spark job — this one on Microsoft Fabric — reads the whole folder into one table. To make that read fast across thousands of tiny files, I did the standard thing: I handed the reader an explicit schema instead of letting it infer one per file. Supplying the schema is a real optimization — the engine can parallelize the read instead of sniffing every file’s types one at a time. It’s the advice in every performance guide.

Then, one day, an upstream change dropped the middle column. The files started arriving with just order_id and amount.

Here’s what I assumed would happen: the job errors, because the file doesn’t match the schema. Or, at worst, region comes through as null and someone notices a column full of blanks. Both of those are fine. Both of those get caught.

Here’s what actually happened: nothing got caught.

Spark maps by position, not by name

When you hand Spark’s CSV reader a schema, it does not line your columns up by header name. It lines them up by position. The header row is read and then essentially ignored — it’s there to be skipped, not to be matched. Column one of the file goes into field one of your schema, column two into field two, and so on, regardless of what the headers say. This is documented, deliberate behavior, controlled by an option (enforceSchema) that defaults to “trust me, apply my schema positionally.”

So when region disappeared from the middle of the file, the columns shifted left. The file’s second column was now amount — but position two in my schema was region. Spark dutifully poured the order amounts into the region field, and left amount to be filled from a column that no longer existed. No type error fired, because the values were strings landing in a string field. No null-flood appeared in an obvious place.

Every row was corrupt, and every row looked completely plausible. Downstream, regional totals were computed from a column that now held order IDs and dollar figures. The dashboards updated. The numbers moved a little. Nobody’s job crashed. That’s the whole horror of it: a missing column should crash, and instead it shifted.

A loud failure is a gift

The thing I keep coming back to isn’t the Spark option. It’s the shape of the failure.

An error is the best possible outcome when something is wrong. An error stops the line, points at the problem, and refuses to hand you a bad answer. The dangerous failures are the silent ones — the ones that pass every check that exists because no check was watching that, and produce output that’s wrong in a way you can’t see by looking. I’d optimized for speed and, without noticing, traded a loud failure I’d have caught in minutes for one that took a week and a suspicious analyst to surface.

The fixes are all about refusing the silence:

• Make the reader check the header. Setting enforceSchema to false tells Spark to validate the file’s header against your schema and complain when they don’t line up — turning the silent shift back into the loud error it should have been. You give up a little of the parallel-read speed; you buy back the failure mode you actually want.
• Merge by name, not by position. Tools built for messy files align columns by header — DuckDB’s read_csv_auto(..., union_by_name=true) will slot each column into the right place by name and fill genuinely-missing ones with null, instead of shifting. On a single machine, for a pile of small files, it’s often faster than the distributed option anyway.
• Assert the shape before you trust it. A one-line check that the columns you expect are present, by name, before the data goes anywhere — the data-engineering equivalent of a smoke detector. Cheap, boring, and it would have caught this on day one.

There’s nothing exotic here. The lesson is the boring, expensive one: the engine did exactly what I asked. I asked it to be fast, and I didn’t ask it to tell me when the file stopped matching my assumptions. Speed I could see. The assumption I couldn’t — until it broke, silently, and kept running.

Now the first thing my file readers do is verify the columns are where I think they are. A pipeline that crashes on a bad file is working. A pipeline that keeps running on one is the thing to be afraid of.

Try it

Drop a CSV to inspect its columns and types; drop a second — or hit Load an example — to diff the headers and watch a dropped column shift everything after it out of alignment, which is exactly what a positional load does without a word. Nothing is uploaded. (Open it full-page.)