SI Database / Journal

Why Intelligent Search Still Needs Exact Words

On this page

The argument for searching by meaning usually begins with a tidy example: a person asks for “winter clothing” and the system finds “insulated jacket.” The relationship is easy to recognize, and an exact keyword match could have missed it. But now imagine searching for a medicine’s product code, a particular invoice, or the name of a little-known musician. An approximate match may be precisely what you do not want.

This is the limitation of treating semantic search as a replacement for ordinary search. Some questions are about concepts. Others are about exact identifiers, and many mix the two. A good retrieval system has to know when flexibility helps and when it introduces confusion.

Two different forms of relevance

Traditional keyword search scores documents partly by their overlap with the terms in a question. Systems such as BM25 take account of the frequency of words in a record and how distinctive those words are across a collection. Vector search, by contrast, compares numerical representations learned by a model. One is particularly useful when precise language is essential; the other can recover related material expressed differently.

Suppose a support engineer searches for “Error E1047 while importing invoices.” A semantic result about invoice-processing failures might provide useful background, but a document that names E1047 explicitly could be the more immediate answer. A system optimized only for broad similarity might rank the wrong page first. That is not a failure of meaning-based retrieval. It is a mismatch between the retrieval technique and the question.

The hybrid approach

Many databases and search products support hybrid retrieval, combining keyword and vector results. The ranking can balance semantic similarity with exact-term matches, and metadata filters can restrict results to a given date, document type, or permission level. The combination is often useful because the relevance of a result has several dimensions. The nearest sentence in a model’s numerical space is not necessarily the record a user needs.

Hybrid does not guarantee a correct answer either. Weights must be tuned, datasets checked, and results evaluated using realistic questions. A search engine may work well on common queries while performing poorly on specialized vocabulary. An evaluation set should therefore include the odd, precise, difficult questions that motivated the project in the first place.

The practical question

Before choosing an engine, make a small inventory of the questions people will ask. Which require literal matches? Which require synonyms? Which depend on permissions or freshness? Which should return no result rather than a speculative one? The answers point toward the appropriate retrieval method more clearly than a list of fashionable product features.

Intelligent search is not a contest between old and new. It is an effort to recognize that words can refer both to ideas and to exact things. Useful systems learn to respect both.

Sources and further reading

← Back to the journal