Every pipeline that loads files has to answer one deceptively small question before it does anything else: have we already loaded this? Get it wrong in one direction and you load the same data twice. Get it wrong in the other and you skip data you needed. Most pipelines answer it with the filename, and the filename is the wrong thing to ask about.

Here’s why it breaks, in both directions.

A source sends you sales_2026_01_10.csv. You load it, record the name, move on. A week later they find a bug in that day’s numbers, fix it, and re-send — same filename, corrected contents. Your pipeline checks its list, sees sales_2026_01_10.csv, says “already loaded,” and skips it. The correction never lands. You are now confidently serving the wrong number, and nothing failed to tell you so.

Now the other direction. A re-run upstream produces the identical data under a new name — sales_2026_01_10_v2.csv, or a fresh timestamp. Your pipeline has never seen that name, so it loads it, and the same rows now exist twice.

Both bugs come from asking about the label instead of the contents. The filename is a name the source chose; it is not the data. The question you actually care about is “have we seen this exact data before,” and there’s a boring, exact answer to it: hash the file’s contents. A content checksum — MD5, SHA-256, whatever — is the same whenever the bytes are the same and different whenever they aren’t, regardless of what the file is called or when it showed up. Rename-proof, clock-skew-proof, rerun-proof.

Once you’re keying on content instead of name, something better falls out of it — you can finally tell apart two cases that filename-dedup smears together:

• Same key, same content hash → a true duplicate. You’ve seen this exact data; skip it.
• Same key, different content hash → not a duplicate. It’s a correction. The source is telling you that the same invoice, on the same day, looks different now than it did before — which is a fact you very much want to keep, not throw away.

Filename dedup can’t distinguish these, so it does the worst possible thing: it treats the corrected re-send as a duplicate and silently discards the fix. Content-hashing turns “did I see this name” into “did I see this data — and if the data changed, is this a redelivery or a revision,” which is the question you were always actually asking.

This isn’t clever work. A checksum column and a small file registry is a boring afternoon of it. But it’s the difference between a pipeline that serves stale corrections for a month and one that can look at two files with the same name and correctly say “that one’s a copy, that one’s a change.” The label was never the truth. The bytes are.

Try it

Drop a file or type text to watch its SHA-256 fingerprint update locally in your browser. Use the buttons to simulate renaming the file or changing a single byte to see how a checksum-based system differentiates them. (Open it full-page.)