The Vector Database Choice Mattered Less Than the Chunking Strategy

We ran a formal bake-off between three vector databases for a RAG project — benchmarking latency, cost at scale, filtering capability, hosting options, all the things you're supposed to compare. It took two weeks. The database we picked ended up contributing, by our own later measurement, almost nothing to the retrieval quality improvement we eventually shipped. The chunking rewrite we did afterward, almost as a side quest, moved retrieval precision from 0.61 to 0.84.

What the bake-off actually told us

All three databases we tested — a managed option, a self-hosted option, and a hybrid search option — performed within a few percentage points of each other on retrieval quality when given the same embeddings and the same chunks. The differences that actually mattered were operational: hosting cost at our scale, how painful metadata filtering was to set up, and how well it integrated with our existing infrastructure. Those are real considerations. They are not retrieval quality considerations, and we'd implicitly treated them as the same thing.

A vector database finds nearest neighbors correctly. Whether those neighbors are useful was decided before the database ever saw them.

What actually moved the needle

Before: fixed 512-token chunks, no overlap, split mid-sentence and mid-table
After:  semantic chunking by section heading, 150-token overlap,
        tables kept intact with their header row attached

Retrieval precision: 0.61 -> 0.84
Vector DB used: same one, unchanged

• Fixed-size chunking was slicing tables apart and splitting arguments mid-sentence, destroying semantic coherence before embedding even happened.
• Switching to section-aware chunking with meaningful overlap fixed most of our precision problem without touching infrastructure.
• Embedding model choice mattered more than database choice — upgrading the embedding model alone added another few points of precision.
• We re-ran the same three-database bake-off after the chunking fix and got the same conclusion: pick based on ops, not retrieval benchmarks.

If you're staring down a vector database decision right now, my honest advice is to spend a day picking something reasonable and boring, then spend the two weeks you were going to spend benchmarking databases on your chunking and embedding strategy instead. That's where the retrieval quality actually lives.