I Built a Mini Search Engine for an Internal Product. The Algorithm Was the Easy Part.

By Muhammad Umar·March 23, 2026·7 min read·Issue #15

It was search inside an internal product. Not a search engine anyone would call a search engine, just the box at the top of the page that the people using the tool all day rely on to find one record among the rest.

The first version worked. A few thousand records, a query that looked at all of them, and results that came back quickly enough that nobody asked how it worked.

It stayed fine for a while. Then it stopped being fine, and not on any particular day. Every record added made searches take a little longer, and the change was small enough each time that there was never a moment where something obviously broke.

By the time it was a problem it had been a problem for months.

I spent a long time after that assuming the fix was a better algorithm. It was not. The algorithm turned out to be the half of this I understood. The half that decided whether the thing scaled was underneath it, in the data.

The version that scanned everything

The original design was the obvious one. A search request arrived, the system looked at every record, scored each one for relevance, sorted the scores, and returned the top few.

It is a completely reasonable thing to build first. It is correct, it is easy to reason about, and at a few thousand records it is genuinely quick. For an internal tool it is also the sensible amount of engineering, because nobody wants a retrieval pipeline in a product whose search box is used forty times a day.

The trouble is the shape of it. Every request does work proportional to the whole dataset, so the cost of a search is tied to how much data exists rather than to how much of it is relevant. A search matching three records costs the same as one matching thirty thousand.

That is fine while the dataset is small and gets worse forever. Growth that is good news everywhere else in the business arrives here as degradation.

Why caching did not save it

The first thing I reached for was caching, because the first thing everyone reaches for is caching.

It helped, and it helped with the wrong requests. Popular searches got quick, because popular searches repeat. But the searches people actually make are mostly unique: a specific name, a partial reference, a phrase with a typo in it. Those miss the cache every time, and each miss still scans everything.

So the average looked better while the experience did not improve for the person doing an unusual search, which is exactly the person who needs search rather than browsing.

Caching hides a cost you are still paying. It does not remove it, and it hides it for the requests that were already cheap.

Splitting the work into two stages

The rebuild came from noticing something about the expensive part. The scoring was expensive because it was thorough: fuzzy matching, several ranking signals, comparisons that had to look at the actual content.²

None of that thoroughness was wasted. It was just being applied to millions of records that were never going to be in the answer.

So the pipeline got split. A first stage that is cheap and generous, using indexes and light filters to cut the dataset down to a candidate set. Then the expensive scoring, run only on what survived.

The expensive stage still costs exactly what it always cost. It simply stopped being asked to do it millions of times.

This is a very old idea in information retrieval, where the two halves are usually called candidate generation and ranking.¹ I did not know that at the time, which is its own small lesson about how much of this is already written down.

The property that mattered most was not the speed on the day. It was that response times stopped tracking the size of the table. Adding records makes the first stage return a slightly different candidate set. It does not make the second stage do more work.

The trade you are making

The first stage is allowed to be wrong in one direction only. It can hand the ranker records that turn out to be irrelevant, because the ranker will drop them. It must not miss a record that should have been in the answer, because nothing downstream can recover it.

That constraint is the whole design of the first stage, and getting it wrong is silent. A missed result does not error. Nobody reports the thing they did not find.

The part I did not see coming

With performance predictable, I expected to be finished. The actual surprise had been there the whole time and had nothing to do with speed.

The same record existed several times, in slightly different forms. Extra spaces. Different capitalisation. An abbreviation in one and the full word in another. A comma. A spelling that was almost right.

To a person these are obviously one thing. To an index they are unrelated strings, so they were being treated as separate entities. Searches returned the same real-world thing three times in different positions, ranked independently, competing with each other for space in the results.

This is worse than it sounds, because it degrades the results twice. The duplicates take slots that should have gone to other matches, and the ranking signals get divided between copies, so the correct answer can score below something less relevant simply because it was split three ways.

Fixing it in the database rather than the algorithm

The tempting fix is to make the matching cleverer, so it sees through the differences at query time. I am glad I did not, because that is expensive work repeated on every single request, forever, to compensate for something that could be settled once.

What I did instead was store the data twice. The raw values stay exactly as they arrived, untouched, because they are what the user gave and what has to be shown back. Alongside them sit normalised values built for searching: case folded, whitespace collapsed, punctuation and accents stripped, abbreviations expanded. The search indexes point at those.

The cost is real. Writing gets more involved, since the normalised form has to be produced and kept in step. Changing the normalisation rules means rebuilding indexes across the whole dataset. The schema is harder to explain to someone new.

What you get back is that the messiness is dealt with once, at write time, instead of being rediscovered on every query by an algorithm asked to be clever about it.

It also made duplicates detectable rather than merely tolerable. Two records that normalise to the same value are visibly the same thing, and can be collapsed instead of competing.

When this is the wrong advice

When your dataset is small and staying small. Scanning everything is correct at a few thousand records. It is simple, it has no candidate stage that can silently drop the right answer, and it will outlive several rewrites of the clever version. Build the obvious thing first. I would build it again.

When normalisation destroys meaning. Case folding is harmless for company names and wrong for identifiers where case is significant. Stripping punctuation ruins part numbers and code snippets. Expanding abbreviations is a guess, and it is sometimes the wrong guess. Every rule you add is a small claim about what two strings mean, and some of those claims are false.

When you should not be building this at all. Search engines that already do candidate generation, fuzzy matching, and ranking exist and are very good. I learned an enormous amount doing it myself and I would not assume that trade is right for everyone. If search is not the thing your product is actually about, use something that already works.

What I would tell myself at the start

Most of the work on this happened before anything shipped, and almost none of it is visible in the result. Days spent reading how other people had solved it. A database design that got redone several times, each version turning up an edge case the last one had hidden. Approaches that looked promising and were thrown away.

The finished thing looks obvious now, which is the usual way with these. It looks obvious because every wrong turn was removed from it.

The thing I would actually tell myself is narrower than a lesson about hard work. Search quality is a property of your data before it is a property of your algorithm. I spent the early weeks trying to score better, and the score was never the problem. The problem was that the same thing was in the database four times wearing slightly different clothes, and no ranking function is clever enough to fix that from the outside.