The Gap Between a RAG Demo and a RAG Product Is Where Projects Die

The demo went great. We pointed a basic RAG pipeline at a folder of clean PDFs, asked it three cherry-picked questions, and got three good answers. Leadership was thrilled. Then we pointed the same pipeline at the real document corpus — eleven years of inconsistently formatted internal wikis, PDFs with tables that didn't parse, and documents that contradicted each other depending on which year they were written — and the whole thing fell apart within a day of real usage.

What actually broke

The chunking strategy that worked fine on clean prose destroyed tables, splitting rows from headers so the retrieved context was numbers with no label attached. Several documents directly contradicted newer ones without any versioning metadata, so the retriever would confidently pull a superseded policy and the model would confidently cite it. And a meaningful chunk of user questions weren't lookup questions at all — they were comparison or reasoning questions that no single retrieved chunk could answer, no matter how good the retrieval was.

A demo proves the mechanism works. Production proves whether your documents and your users were ever going to cooperate with that mechanism.

The actual fixes took months, not days

• Document preprocessing to normalize tables into retrievable text, not an afterthought bolted onto chunking.
• A metadata layer tracking document recency and supersession, so retrieval could be filtered or re-ranked by freshness.
• A separate triage step classifying whether a question was answerable by retrieval at all, before burning a generation call on a question that needed synthesis across documents.
• An eval set built from real user questions logged during a pilot, not the three cherry-picked demo questions.

Budget for the gap, not the demo

The project didn't die because RAG doesn't work. It nearly died because the timeline and budget were set based on how long the demo took, and nobody flagged that the demo and the product were solving almost entirely different problems. The demo answers 'can retrieval find relevant text.' The product has to answer 'can this system handle contradictory, poorly structured, constantly changing documents being asked messy real questions by people who don't know or care how RAG works.' Those are different projects with the same name, and only one of them fits in a sprint.