Advanced RAG Patterns: Hybrid Search & Re-ranking

Why naive RAG plateaus

A basic RAG pipeline — embed the query, do a nearest-neighbor search over a vector index, stuff the top-k chunks into the prompt — works, but it has a predictable ceiling. Pure embedding similarity misses exact matches on names, codes, and rare terms that happen to sit far apart in embedding space from their semantic paraphrase. It also retrieves chunks independently, with no step that checks whether the retrieved set actually answers the query well as a group, and it trusts the user's literal query phrasing even when that phrasing is a poor match for how the answer is actually written in your documents.

Advanced RAG patterns address each of these gaps as separate, composable stages rather than trying to fix everything with a better embedding model alone.

Hybrid search: combining dense and sparse retrieval

Dense retrieval (embeddings + vector similarity) captures semantic meaning — it finds passages that mean the same thing even with different words. Sparse retrieval (classic algorithms like BM25, built on term frequency statistics) captures exact lexical matches — it's excellent at finding a specific product code, acronym, or proper noun that an embedding model might blur into a generic semantic neighborhood.

Hybrid search runs both in parallel and merges the results. The two ranked lists use different, non-comparable scoring scales (cosine similarity vs. a BM25 score), so you can't just average them directly. The standard fix is Reciprocal Rank Fusion (RRF): for each document, take its *rank position* in each list (not its raw score), convert that rank to a score (typically 1 / (rank + constant)), and sum across both lists. This sidesteps the scale-mismatch problem entirely and tends to produce a robust, blended ranking.

Cross-encoder re-ranking

Embedding similarity search works by comparing a query vector against pre-computed document vectors independently — the query and document never actually interact with each other during scoring, which is exactly what makes it fast enough to search millions of vectors. A cross-encoder re-ranker gives up that speed for a later, smaller stage: it takes the query and a candidate document together, as a single joint input, and scores their relevance directly, letting the model attend to specific overlaps and relationships between query and document text that independent embeddings simply can't represent.

The standard pattern: retrieve a wider candidate set (say the top 20-50 chunks) cheaply with hybrid search, then re-rank just those candidates with a cross-encoder, and keep only the top 3-5 for the prompt. This two-stage retrieve-then-rerank approach consistently outperforms either stage alone, because each stage is doing the job it's actually good at.

Embedding similarity vs cross-encoder re-ranking

Query rewriting: fixing the question before you search

Users don't phrase questions the way your documents are written. A terse, ambiguous, or conversationally-dependent query ("what about the enterprise one?") often retrieves poorly even with a perfect retrieval pipeline, because the retrieval step has no idea what 'the enterprise one' refers to without the preceding conversation.

Query rewriting uses an LLM to transform the raw query before it ever reaches the retriever: resolving references from conversation history, expanding abbreviations, or decomposing a compound question into sub-questions that are each retrieved separately. A related technique, multi-query retrieval, generates several different phrasings of the same underlying question, retrieves for each, and merges the results — catching relevant chunks that any single phrasing might have missed due to vocabulary mismatch.

What's next

Hybrid search works because dense and sparse retrieval fail in different, complementary ways. The next lesson goes deeper into that specific comparison — semantic vs keyword search — including the cases where plain keyword search still beats embeddings outright.