The single most useful thing I learned in a decade of building things is not a technique. It’s a suspicion.

I trace it to a specific moment. I’d built a search engine by hand, typed in queries, and gotten back lovely, relevant-looking results. It felt finished. Then I benchmarked it properly against a test collection with real relevance judgments, and it scored about 0.01 precision. The engine that delighted me by eye was, by measurement, returning almost nothing right. The gap between “looks great to me” and “is great by measurement” was the widest I’d ever seen in my own work — and closing that gap was the entire education. The building had been the easy part. Not believing the demo was the hard part, and the important one.

That reflex got built by repetition, one humbling number at a time.

A decade of impressive numbers that were lies

The GPU port I refused to call a speedup. I moved a collision detector onto a GPU and wanted, badly, to report a win. But my timer resolution was milliseconds and the numbers I was comparing were a small number of milliseconds — noise. I couldn’t cleanly prove the GPU was faster, so I didn’t claim it was. Saying “I can’t measure this well enough to make the claim” is a skill, and a deeply unnatural one, because the CUDA in the title is right there doing the implying for you.

The deep-learning model that got an honest 61%. My capstone classified brain MRIs and landed only modestly above a majority-class baseline, while published work reached the 90s. The disciplined move was not to chase their number — it was to distrust it if I ever hit it. A suspiciously good result on hard, small, messy data is the first thing to audit, not the thing to celebrate. If I’d gotten 95%, my first job would have been to go find the leak.

The classifier whose headline accuracy meant nothing. I once modeled a lopsided public dataset where about 99% of cases went one way. A model reported “80% accuracy” on a held-out slice and looked like a success — until you noticed that slice still carried the real 99%-to-1% ratio, so the number was just the base rate leaking through a split I’d botched. It was an impressive figure sitting on top of a broken evaluation. The model had learned nothing and the metric applauded it anyway.

The text classifier that learned the wrong thing. Another one scored a comfortable 0.81 accuracy and a much worse 0.59 macro-F1 — and when I looked at why it was right, it had learned to detect the register of the writing, not the truth of the claims. It flagged anything that read like a hasty social-media post and trusted anything that read like a wire report. A style detector wearing a truth-detector’s label, and only a per-class metric plus an explanation exposed it. The accuracy number would have let me ship it.

What the pattern taught me

Every one of those has the same shape. The demo, or the top-line metric, said great. Something more honest — a benchmark, a per-class score, an explanation, a properly-resolved timer — said no. And the disagreement was always the most valuable thing in the project, because it pointed straight at what I actually didn’t understand yet.

Out of enough of those, a small working epistemics crystallized, and I use it on everything now:

• The number that looks best is the one to check hardest. Delight is a signal to audit, not to ship.
• Pick the metric that can embarrass you. Accuracy can’t embarrass a model on imbalanced data; macro-F1 can. A benchmark can embarrass you; eyeballing never will. Choose the measure with the capacity to say you’re wrong.
• Ask not just whether it’s right, but why. A right answer for the wrong reason — style instead of truth, base rate instead of signal — will betray you the instant the distribution shifts. Correctness you can’t explain is borrowed, not owned.
• If you can’t measure it cleanly, don’t claim it. An unverifiable win is not a win. It’s a story you’re telling yourself.

None of this is exotic. It’s just the scientific method with the lab coat off. A demo is a hypothesis that happens to flatter you; a benchmark is the experiment that doesn’t care how you feel. The whole discipline is preferring the second to the first — falsification over confirmation — even, especially, when the first one is yours.

The present-day form

The same suspicion is what I bring to AI now. The tasks you can safely hand a model are the ones where a deterministic check exists to embarrass it when it’s wrong; the dangerous ones are where no check exists and a plausible answer sails straight through. A demo is precisely the thing built to look right to someone who wants it to — which is why I trust it least of all.

I’m not smarter than my demos. I’ve just been fooled by enough of them to stop trusting the ones that flatter me. That suspicion — of my own good-looking results, first and hardest — is the most scientific thing I do.