January 6, 2026 · 5 min read
AI-Powered Search Was Worse Than Keyword Search for This Specific Use Case
We swapped a boring keyword search for a semantic AI search on a parts catalog and watched conversion drop, because precision mattered more than the AI's sense of relevance.
Semantic search is the obvious upgrade over keyword search in most pitches I've seen, and for a lot of use cases that's true. It wasn't true for a client selling industrial replacement parts, where a user searching for a six-character SKU needs exactly one result, not seven plausibly related ones.
Why the swap made things worse
A customer typing a part number already knows what they want. Semantic search, by design, tries to understand intent and surface conceptually related results — which is exactly the wrong behavior when the user's intent is maximally specific. We saw queries for an exact part number return five near-matches ranked by embedding similarity instead of the one exact match at the top, because the embedding model treated 'similar-sounding part number' as semantically close.
Semantic search assumes the user wants to be understood. Sometimes the user just wants to be found exactly, and understanding gets in the way.
What the data showed
Click-through on the top result dropped from roughly 70% under keyword search to 45% under semantic search, and support tickets about 'can't find my part' rose noticeably in the following month. Conversion on search-originated sessions dropped by about 12%. None of that showed up in the pre-launch testing, because the test queries we used were conversational, not the terse alphanumeric strings real customers actually typed.
- Exact-match queries (SKUs, model numbers, order IDs) should route to literal matching first, with semantic search as a fallback only when exact match returns nothing.
- Test search quality with the query patterns your actual users type, not the query patterns a demo script types.
- A hybrid approach — keyword-first with semantic re-ranking only on the no-results case — recovered most of the lost conversion while keeping semantic search's benefit for genuinely exploratory queries.
The fix wasn't abandoning AI search, it was recognizing that "more intelligent" and "more useful" aren't the same thing when the user already knows precisely what they want.