Semantic Search
Searching by meaning rather than by words — which finds what keyword search misses, and misses what keyword search finds.
When not to use it
- When users search for exact strings. Identifiers, codes, names. Semantic search will find things about them and not them.
- When your corpus is small. Under a few thousand documents, the whole apparatus may be more machinery than the problem needs.
- Alone, on any real corpus. Nearly every production system that starts semantic-only ends up hybrid.
- When your domain vocabulary is unusual and you're using an off-the-shelf embedding model. It doesn't know your words, and "similar meaning" quietly degrades.
Reach for something else instead
- BM25 / keyword search — decades old, still wins on exact terms and rare words, costs nothing.
- Hybrid with RRF — what you almost certainly want. Both retrievers, ranks fused.
- Metadata filtering — often the actual need. "Similar, from this customer, last 90 days" is a filter problem, not a search problem.
- Fine-tuned embeddings — when domain vocabulary is the bottleneck and you have labelled pairs.
Read more on the blog
- Beyond vector search: how RAG actually works in 2026RAG stopped being "vector database plus a language model" a while ago. Here's how retrieval-augmented generation actually works now, chunking, embeddings, reranking, knowledge graphs, and agentic retrieval, and where each piece quietly breaks.
- Why AI thinks hot and cold mean the same thingWord embeddings score "accept" and "reject" at 0.73 similarity. The reason is the idea embeddings are built on, and it explains why cosine similarity is a weaker signal than most systems treat it as.
- What are embeddings? How AI turns meaning into numbersWhen a search returns the right result without sharing a single keyword, when a chatbot pulls the relevant document, when a store recommends something that just fits, the technology underneath is almost always embeddings. They rest on one powerful idea: turn meaning into geometry, so that similar things sit close together in space and similarity becomes something a computer can measure.
- Do you actually need a vector database?Vector databases became the default first purchase for anyone building with AI. For most projects they're the wrong first move, here's how to tell whether yours is the exception.
The full account
Why keyword search fails, precisely
For fifty years, search meant matching words. You type a query, the system finds documents containing those words, ranked by a formula that rewards rare terms and penalises long documents. That formula is BM25, it dates to the 1990s, and the striking thing about it in 2026 is that it has not gone away. It is still the baseline every new method is measured against, and it still wins on a large class of queries. Any honest account of semantic search has to start by admitting what it is trying to replace actually works.
The failure of keyword search is specific, not general. It fails when the searcher and the document use different words for the same idea. Someone types "how do I get my money back" into a corpus that files everything under "reimbursement eligibility." The concepts are identical; the strings share not one content word. BM25 sees no match because, mechanically, there is none. This is the vocabulary mismatch problem, and it is not a rare edge case — it is what happens every time a real person searches a corpus written by specialists, which is most enterprise search, most documentation, most support.
Semantic search is the attempt to match the idea instead of the string.
What "meaning" means here
The mechanism is worth stating without mysticism, because the mysticism is where the disappointment comes from.
A neural network — an embedding model — is trained to convert a piece of text into a list of numbers, a vector, positioned in a high-dimensional space such that texts with similar meaning land near each other. "Get my money back" and "reimbursement eligibility" end up close, because the model has seen enough text to learn they travel in the same company. You embed every document once, store the vectors, embed the query at search time, and return the documents whose vectors are nearest. Nearness is usually cosine similarity — the angle between two vectors.
That is the whole idea, and its power is real. But notice the quiet assumption: "similar meaning" is whatever the model learned it to be, from whatever text it was trained on. A general-purpose embedding model learned meaning from general text. Hand it a corpus of part numbers, internal codenames, or a specialist vocabulary it never saw, and "similar meaning" quietly degrades toward "similar-looking." This is the single most common reason semantic search dazzles in the demo and disappoints on the actual documents — the demo uses ordinary language the model knows, and the real corpus doesn't.
The result that made it standard, and the exception that proves the point
Dense retrieval existed as an idea for years before it worked well enough to trust. The paper that changed that was Karpukhin and colleagues' Dense Passage Retrieval in 2020. Using a simple dual-encoder — one network embeds questions, another embeds passages, both trained so that a question lands near its answer — they beat a strong BM25 system by 9 to 19 percentage points on top-20 retrieval accuracy across open-domain question-answering benchmarks. On Natural Questions the gap was 78.4% against BM25's 59.1%. That margin is what moved dense retrieval from research curiosity to production default, and it is why nearly every retrieval stack built since assumes embeddings somewhere in the pipeline.
But the same paper contains the result that should keep everyone honest, and it is the one nobody quotes. On SQuAD, BM25 won — 68.8% against DPR's 63.2%. The reason is exactly the thesis of this page. SQuAD's questions were written by people looking at the passage, so they reuse the passage's own words; the vocabulary already matches, and there is nothing for semantic search to add — while its weakness at exact term-matching quietly costs it. The method that wins by 19 points on one dataset loses on another, and the thing that flips the result is whether the query and the answer share vocabulary. That is not a footnote. That is the whole trade-off, printed in the founding paper's own results table.
One more number from that paper, because it reframes the cost: DPR trained on just 1,000 examples already outran BM25. The barrier to semantic search was never enormous training data. It was having a model at all.
| Keyword (BM25) | Semantic (dense) | |
|---|---|---|
| Matches | strings | meaning |
| Wins on | exact terms, codes, names, rare words | paraphrase, synonyms, natural questions |
| Fails on | vocabulary mismatch | exact identifiers, unusual domain vocabulary |
| DPR vs BM25, Natural Questions | 59.1% | 78.4% |
| DPR vs BM25, SQuAD | 68.8% | 63.2% |
| Cost | an inverted index | a model, a vector store, and its ANN index |
Why the answer is almost always "both"
The table above is the argument for hybrid search, and it is the single most useful practical conclusion in this whole topic. The two methods fail on different queries. Keyword search fails on paraphrase; semantic search fails on exact strings. A system that runs both and combines their results fails on neither, and that is not a marginal improvement — on most real corpora it beats either method alone by a wide margin.
The combining step has a standard, unglamorous solution: Reciprocal Rank Fusion. The problem it solves is that BM25 scores and cosine similarities are not on the same scale — you cannot add them, because a BM25 score of 12 and a cosine similarity of 0.7 mean nothing to each other. RRF sidesteps this entirely by ignoring the scores and using only each document's rank in each list. A document ranked highly by both methods rises to the top; the arithmetic is trivial, it needs no tuning, and Cormack and colleagues showed in 2009 that it beats more sophisticated learned combiners. The most robust piece of a modern retrieval stack is also one of its simplest.
The constraint that shapes everything downstream
There is one structural fact about semantic search that explains most of its remaining weaknesses: the query and the document never meet. They are embedded independently — the document at indexing time, the query at search time — and compared only as finished vectors. The model never gets to look at the query and the document together and ask whether one actually answers the other.
This is exactly what makes semantic search scale. Because passages are embedded in advance, searching millions of them is a nearest-neighbour lookup, not millions of comparisons — and even that lookup is approximate (the HNSW algorithm and its relatives trade a sliver of recall for enormous speed). But the same independence is the accuracy ceiling. A passage can be maximally similar to a question and still not answer it, because similarity and answerhood are different relations — questions and their answers often use different words and different structure. This gap is where a great deal of unexplained retrieval disappointment actually lives, and it is the reason reranking exists: a reranker looks at the query and document together and re-scores the shortlist, buying back accuracy the independent embeddings gave away for speed.
What to actually build
The advice that follows from all of this is unglamorous and worth following. Do not replace keyword search — add to it, and fuse the two with RRF. Do not trust an off-the-shelf embedding model on a specialist corpus without checking it on your actual documents and vocabulary. Expect a shortlist from retrieval and a reranker to sharpen it. And remember what determines the winner on any given query: whether the searcher and the document share words. When they do, keyword search was already enough. When they don't, semantic search is the only thing that works. A system that assumes it is always one or always the other is a system that fails half its queries — and the founding paper told you so in 2020.
Further reading
- Karpukhin et al. (2020), Dense Passage Retrieval for Open-Domain Question Answering — the paper that made dense retrieval standard.
- Robertson & Zaragoza (2009), The Probabilistic Relevance Framework: BM25 and Beyond — the keyword baseline that keeps refusing to lose.
- Cormack et al. (2009), Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods — the combiner behind hybrid search.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Replacing keyword search rather than adding to it, then rediscovering exact-match queries the hard way.
- Using a general embedding model on a specialist corpus and blaming the retrieval system for the results.
- Ignoring the asymmetry between short queries and long passages.
- Assuming high similarity means the passage answers the question. It means the passage resembles the question.
- Combining BM25 and cosine scores by adding them. They aren't on the same scale — fuse ranks, not scores.
At a glance
Often compared with
Where this sits
8 concepts come first. Understanding it opens up 5 more.
Computed from the prerequisite graph, not assigned. How this works