RAG Is Not a Silver Bullet: Where Retrieval Actually Breaks Down

RAG gets pitched as the fix for hallucination: ground the model in real documents, and it'll stop making things up. That's true in the demo, where the knowledge base is small, clean, and the test questions happen to map cleanly onto a handful of well-chunked documents. In production, with a real knowledge base that grew organically over three years, RAG has its own failure modes — and they're much harder to spot than a model confidently inventing a fact, because they look like the system working correctly right up until it isn't.

Chunking is where most of the damage happens

A fixed-size chunking strategy — split every 500 tokens, say — has no idea where a document's actual logical boundaries are. It'll cut a table in half, separate a caveat from the claim it qualifies, or split a step-by-step procedure across two chunks that get retrieved independently and recombined out of order. The retrieval system then dutifully returns the broken fragment, ranks it as relevant (because the keywords match), and the model generates a confident, fluent, subtly wrong answer built on a half-sentence of context.

A RAG system doesn't hallucinate less when the retrieval is bad — it hallucinates differently, and the output still looks grounded because it's citing a real, retrieved chunk.

Retrieval ranking optimizes for the wrong thing by default

Vanilla vector similarity search ranks by semantic closeness, not by whether a chunk actually answers the question, and those two things diverge more often than most teams expect. A chunk can be semantically adjacent to a query — same topic, similar vocabulary — without containing the specific fact the query needs. This is exactly the gap that hybrid search (combining vector similarity with keyword/BM25 matching) and re-ranking models exist to close, and it's also exactly the step most teams skip when they ship a RAG pipeline built from a tutorial's default configuration.

Stale indexes are a silent, compounding failure

A vector index built once at launch and never refreshed doesn't fail loudly when the underlying documents change — it just keeps confidently serving outdated information, and the model has no way to know its context is stale. This is the RAG equivalent of a dashboard built on a pipeline that silently stopped updating six weeks ago: everything looks fine until someone notices the numbers haven't moved and realizes nobody's been checking.

• Chunk with document structure in mind (headings, tables, list boundaries), not just a fixed token count.
• Add a re-ranking step after initial retrieval — vector similarity alone systematically misranks for factual specificity.
• Build index freshness monitoring, not just index-build automation — know when the last refresh happened and alert on drift.
• Evaluate retrieval quality separately from generation quality — a bad answer built on good retrieval and a bad answer built on bad retrieval need completely different fixes.

Query → embed → vector search (top-k) → [often stops here]
                                              │
                                              ▼
                              + re-rank by relevance to actual query
                              + filter on document freshness
                              + structure-aware chunk boundaries
                                              │
                                              ▼
                                   Context passed to generator

None of this means RAG doesn't work — it's still the right default architecture for grounding a model in a private knowledge base. It means the gap between a RAG demo and a RAG system that holds up under real traffic, a messy document set, and six months of content drift is mostly invisible until you go looking for it, and by then it's already producing confidently wrong answers that look exactly like the system working as intended.