RAG Pipelines
The problem RAG solves
A language model's knowledge is frozen at its training cutoff. Ask it about last week's earnings call, your company's internal docs, or a regulation that changed this year — and it will either make something up or tell you it doesn't know.
Retrieval-Augmented Generation (RAG) is the solution: instead of relying on the model's parametric memory (weights), you give it access to a knowledge base at inference time. The model retrieves relevant documents, reads them in its context window, and generates an answer grounded in actual source material.
This is now the dominant architecture for production AI applications — customer support bots, internal knowledge assistants, legal research tools, and code documentation systems all use some form of RAG.
The three-stage pipeline
Every RAG system has the same three stages, regardless of how sophisticated it gets:
Stage 1 — Indexing (offline): Your documents are chunked into pieces, each chunk is converted to an embedding vector, and all vectors are stored in a vector database. This happens once (and whenever your docs update).
Stage 2 — Retrieval (at query time): The user's question is embedded using the same model. A similarity search finds the k most semantically similar chunks in the vector database.
Stage 3 — Generation: The retrieved chunks are injected into the prompt alongside the question. The LLM reads the context and generates an answer based on what it was given.
Note
Chunking: the most underrated decision
How you split your documents matters more than almost any other RAG decision. Chunks that are too small lose context. Chunks that are too large waste context window space and dilute the relevant signal.
Common strategies:
Fixed-size chunking — split every N tokens with a K-token overlap. Simple and fast, but cuts across sentences arbitrarily.
Semantic chunking — split at paragraph or section boundaries. Respects natural document structure.
Recursive character splitting — try to split at paragraphs first, then sentences, then words, always keeping chunks within a target size. Used by LangChain's default splitter.
A good starting point: 512 tokens per chunk, 64-token overlap.
Vector databases
FAISS is great for prototyping but production RAG systems typically use a dedicated vector database that handles persistence, filtering, and scale:
Pinecone — fully managed, fast, supports metadata filtering. Most popular for quick production deployments.
Weaviate — open-source, supports hybrid search (vector + keyword). Good for complex schemas.
Qdrant — open-source, Rust-based, very fast. Great self-hosted option.
pgvector — PostgreSQL extension. Best choice if you're already on Postgres and your scale is modest (under ~1M vectors).
All of them take embeddings in and return the k most similar vectors for a given query embedding.
Hybrid search: keywords + vectors
Pure semantic search misses exact keyword matches. If someone asks "what does RFC 2616 say about cache-control?", a vector search might return documents about HTTP caching in general rather than the specific RFC.
Hybrid search combines dense vector search (semantic) with sparse keyword search (BM25 or TF-IDF), then fuses the rankings. Most production RAG systems use hybrid search with a Reciprocal Rank Fusion (RRF) step to merge the two ranked lists.
As a rule: use pure vector search first to prototype. Switch to hybrid when you notice retrieval gaps for specific terms, names, or identifiers.
Note
Advanced patterns
HyDE (Hypothetical Document Embeddings): Generate a hypothetical answer to the query first, embed that, and retrieve against it. Works surprisingly well for queries phrased as questions.
Re-ranking: After retrieving the top 20 chunks by vector similarity, pass them through a cross-encoder re-ranker (like Cohere Rerank or a local model) to get a more accurate top-5. The retrieval step is fast; the re-ranking is more accurate.
Multi-query retrieval: Generate 3-5 variants of the user's query using an LLM, retrieve for each, deduplicate, and merge. Catches relevant documents that a single query phrasing might miss.
What's next
RAG is the retrieval layer. AI Agents take this further — they don't just retrieve and generate, they decide which tool to call, execute actions, and loop until the task is done. The next lesson covers how agents work and what it actually takes to build one that works in production.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.