I built a search engine once. Not configured one — built one, from an empty file: an inverted index, TF-IDF weighting, cosine ranking, a small boolean query parser. It was a guided grad-school assignment, worked step by step, but its whole point was the constraint: build every piece yourself before reaching for a library, so you learn what’s under the ones you keep reaching for.

Two things from that project stuck with me for years, and neither was the ranking math — which is the part everyone assumes is hard. One was about data structures. The other was about the difference between a demo and a measurement.

A DataFrame is the wrong shape for an index

My first instinct was the comfortable one. I had a corpus of around 40,000 news articles and I wanted a document-term matrix, so I built one as a pandas DataFrame — documents as rows, terms as columns, weights in the cells — and grew it the obvious way, filling cells in as I processed each document.

It died. Not slowly — it hit a memory wall and stopped, on a corpus that isn’t even large by any serious standard.

The mistake is easy to see in hindsight and easy to make in the moment: a document-term matrix is almost entirely zeros. Most terms don’t appear in most documents. A dense DataFrame dutifully allocates a cell for every one of those zeros — millions of them — because storing every position is exactly what a dense matrix is for. I was spending nearly all my memory recording the absence of things.

The fix wasn’t to optimize the DataFrame. It was to throw it out for the structure that matches the data’s actual shape: an inverted index. Plain dictionaries — term -> {doc_id: frequency}. You only store the postings that exist; the zeros simply aren’t there because you never write them down. Indexing went from “falls over” to “fine,” and nothing about the algorithm changed. Only the primitive did.

Same cosine math — the inverted index just refuses to store, or scan, the zeros.

There’s a companion lesson hiding in the search step. My first ranking function scored a query by computing cosine similarity against every document in the corpus — a full pass over all 40,000 rows, for every query. The postings version only touches documents that contain at least one query term: you walk the postings lists for the query’s terms and score only those. On a real corpus that’s the difference between touching everything and touching a handful — and that, not the cosine formula, is the entire reason inverted indexes exist. The ranking math is identical either way. The index is what makes it tractable.

I think about this every time I’m tempted to grab the structure I’m comfortable with instead of the one that fits the problem: the shape of the data should pick the tool, not your habits.

The demo looked great. The benchmark said 0.01.

Here’s the part that actually humbled me.

Once it worked, I typed in queries and got sensible, relevant-looking results. Search “international travel,” get travel articles. It felt done. If I’d stopped there — the way a demo always lets you stop there — I’d have called it a success and moved on.

Then I benchmarked it properly. There are classic information-retrieval test collections that ship with a set of queries and human relevance judgments: the answer key of which documents are actually relevant to each query. You run your engine over the queries, compare against the judgments, and compute precision and recall. Real numbers, not vibes.

My precision was about 0.01.

Not “needs tuning” bad. Near-zero. The engine that gave me lovely results when I eyeballed it was, by measurement, returning almost nothing relevant. The gap between “looks great in a demo” and “measurably worthless” was the widest I’d ever seen in my own work, and closing it — figuring out why — turned out to be the actual project. The building had been the easy part.

I never fully nailed it at the time, but I traced most of it to three things, and the three things are the useful part:

• Query expansion was quietly poisoning the queries. I’d added WordNet-based expansion — for each query word, tack on its synonyms. “International travel” helpfully became external, outside, move, journey, trip… In a demo, expansion looks clever. In the benchmark it flooded every query with loosely-related noise and buried the actual signal.
• A hard similarity cutoff was strangling recall. I’d set a cosine threshold below which results were dropped. It made the demo look clean and quietly threw away most of the correct answers.
• The worst one: my document IDs and the answer key’s IDs were different numbering systems. I’d re-indexed documents to my own sequential counters while loading, but the relevance judgments referenced the corpus’s original document numbers. So even when my engine found the right document, I was comparing my ID for it against their ID for it — and they never matched. A quiet, off-by-a-whole-numbering-scheme bug that no amount of eyeballing would ever surface, because eyeballing never checks IDs.

That last one is the lesson under the lesson. A demo tests whether the output looks right to a human who wants it to look right. A benchmark tests whether it is right against something that doesn’t care how you feel. When the two disagree this violently, the benchmark is telling the truth, and the disagreement is a map straight to a bug you couldn’t otherwise see.

Why build the boring thing by hand

You’d never ship this. You’d use a real search library, or a vector database, and you’d be right to.

But building it once, from empty, is the only way I actually understood what those libraries do: an inverted index, a weighting scheme, a ranking function, and — if you’re honest — a way to measure whether any of it works. It’s easy now to bolt retrieval onto an LLM and call it done because the demo answers your test question. This project is why I don’t trust that feeling. Retrieval that looks right and retrieval that measures right are different things, and the only way I learned to tell them apart was to build the boring version and watch it score 0.01 while smiling at me.

The demo is not the system. It never was. It’s just the part that’s easy to stop at.

---

Companion code: ramwise-examples/tiny-search — the inverted index, the postings-only scoring, and the benchmark that humbles the demo.