Hybrid Search vs Vector Search for RAG

Vector search is good at meaning. It is not reliably good at invoice numbers, error codes, product names, dates, or a three-letter acronym that appears once in the corpus.

Keyword search has the inverse profile. It rewards exact terms and struggles when the question and document use different language. Production RAG usually receives both kinds of query, often in the same sentence.

That is the case for hybrid search.

The three retrieval modes

Vector search turns queries and chunks into embeddings and ranks them by similarity. Keyword systems such as BM25 use term occurrence and rarity. Hybrid retrieval runs both and fuses their ranked lists.

If you are still selecting storage, see the broader vector database comparison. The retrieval experiment matters more than the vendor label.

Fuse ranks, not raw scores

Cosine similarity and BM25 scores are not calibrated to each other. Adding them directly gives one system accidental control based on its score range.

Reciprocal Rank Fusion (RRF) combines positions instead:

RRF(document) = Σ 1 / (k + rank_in_list)

The constant k reduces the dominance of the first few positions. A document that ranks well in both lists rises without requiring score normalization.

This is a strong baseline, not a universal winner. You still choose how many candidates each retriever contributes and whether one list needs more weight.

Classify your query population

Build an evaluation set with explicit query types:

identifier     "What does error PAY-104 mean?"
exact phrase   "Where is 'authorized processor' defined?"
paraphrase     "Can a contractor view production dashboards?"
multi-concept  "Retention rules for EU customer exports"
misspelling    "cancle an anual plan"

Report recall@K for each category. One aggregate number can hide vector search failing every identifier query while doing well on abundant paraphrases.

Use the scorecard from RAG Evaluation: Metrics for Retrieval and Answer Quality, and keep the generation prompt fixed during the retrieval comparison.

A useful experiment matrix

Test at least these configurations:

Measure recall@5, recall@20, MRR, latency, and cost. Variant D tests whether a wider first stage plus RAG reranking earns its extra latency.

Inspect wins and losses. If hybrid improves average recall by recovering identifiers but hurts a critical policy category, the implementation needs tuning—not a celebratory average.

Filters belong inside the retrieval plan

Tenant, permission, language, product, and effective-date filters are not cleanup after retrieval. Apply them before or during candidate selection so unauthorized or stale documents never become prompt context.

Approximate vector indexes can behave differently under restrictive filters. Test the exact filter strategy supported by your engine: pre-filter, post-filter, or hybrid. A top-K result containing three permitted documents is not equivalent to searching the permitted corpus for its true top K.

When vector-only is reasonable

Vector-only retrieval can be enough when:

• the corpus and queries are consistently natural language;
• exact identifiers are rare or handled by a separate lookup;
• evaluation shows no critical keyword-only cases;
• latency or infrastructure simplicity has a hard budget.

Keyword-only can also be correct for code catalogs, legal discovery with required phrases, or small structured corpora. “Use hybrid” is a default hypothesis, not a law.

Retrieval decision checklist

• The dataset labels identifier, phrase, paraphrase, and multi-concept queries.
• Keyword, vector, and hybrid baselines use the same corpus and filters.
• Fusion operates on ranks or calibrated scores.
• Metrics are segmented by query category.
• Permission and freshness filters run before context assembly.
• Latency and cost are recorded with relevance.
• A reranker is added only after first-stage recall is measured.

Sources

• Microsoft: Hybrid search using vectors and full text
• Microsoft: Relevance, BM25, vector ranking, and RRF
• Elasticsearch: Reciprocal rank fusion