Why Vector Search Alone Fails in Production RAG

Pure vector search is usually the first thing developers build when implementing RAG. It handles semantic matches like 'how to get a refund' mapping to 'cancellation policy' gracefully. But in production, pure dense retrieval creates subtle, maddening blind spots.

When a customer searches for an exact part number, an error code like 'ERR_403_AUTH', or a specific employee name, dense vector embeddings often fail because they map concepts to fuzzy regions rather than preserving exact token matches.

The limits of dense embeddings

Dense vector embeddings compress passages into semantic vectors. While this captures general meaning, it destroys precision on unique identifiers, timestamps, and low-frequency keywords. If two distinct product models differ by one letter, their embeddings may sit practically on top of each other.

The standard fix: Hybrid Search + Re-ranking

1. 01Sparse retrieval (BM25 / full-text search) to capture exact keyword matches, SKUs, and acronyms.
2. 02Dense retrieval (vector embeddings) to capture semantic concepts and synonyms.
3. 03Reciprocal Rank Fusion (RRF) to merge both candidate lists into a single balanced set.
4. 04Cross-encoder re-ranking on the top 20-30 results to compute true relevance before prompting the LLM.

Hybrid search costs an extra query step but fixes over 80% of RAG retrieval misses.

What to optimize first

BM25 keyword search

Catches exact matches, codes, and identifiers

Dense vector search

Catches intents, fuzzy phrasing, and paraphrasing

Cross-encoder re-ranker

Scores true query-passage alignment at inference

Metadata filtering

Prunes scope by user role, date, or tenant first