Semantic Search Was Overpromised on Our Support Docs Project

The pitch was straightforward: keyword search was missing obvious matches because users phrased queries differently than the docs did, and semantic search would close that gap by matching meaning instead of exact words. We shipped it, measured actual user success rate — did they click a result that resolved their ticket without escalating to a human — and it dropped, from 61% to 54%.

Where semantic search actually struggled

Support queries are often short, specific, and contain exact technical terms — an error code, a specific setting name, a product SKU. Semantic search is strong at matching paraphrased intent and weak at exact-term precision, because embeddings compress exact strings into the same vector space as their rough meaning, and 'close in meaning' is sometimes exactly the wrong result when a user typed an exact error code expecting an exact match.

Semantic search is excellent at understanding what you meant. A lot of support queries don't need that — they need the one document with that exact error code in it.

What actually fixed it

• Hybrid search combining keyword/BM25 scoring with semantic similarity, rather than replacing one with the other.
• Boosting exact matches on error codes and technical identifiers to the top regardless of semantic score.
• Keeping keyword search as the primary path for queries matching a recognizable technical-term pattern, with semantic search as a fallback for conversational queries.
• Re-measuring on the actual success metric, not just 'relevance' as judged by a reviewer skimming results — the gap between those two metrics was most of the surprise.

The hype cycle oversold a universal replacement

Semantic search is a real improvement for a real class of queries — anything conversational, anything where the user's words genuinely don't match the docs' vocabulary. It is not a universal upgrade over keyword search, and treating it as a drop-in replacement cost us three weeks of degraded support outcomes before the metric told us what was actually happening. Hybrid, with the keyword path still doing real work, is what we should have shipped from day one.