Building Production RAG Pipelines: Beyond the Hello World Tutorial

Every RAG tutorial shows you the same thing: chunk some documents, embed them, store in a vector database, retrieve and prompt. It works in a notebook. It falls apart in production.

After building RAG systems that handle millions of documents for enterprise clients, here is what the tutorials don’t tell you.

The Chunking Problem is Harder Than It Looks

The default advice is to chunk documents into 512-token pieces with some overlap. This works poorly for most real-world documents.

The problem is that meaning doesn’t respect token boundaries. A paragraph about quarterly revenue split across two chunks loses context in both. A table of data chunked mid-row becomes useless.

What actually works:

Semantic chunking — split on meaning boundaries (paragraphs, sections, list items) rather than token counts. Libraries like semantic-chunker do this reasonably well.

Hierarchical chunking — store both summary-level and detail-level chunks. Retrieve summaries first, then fetch detail chunks for the most relevant summaries. This is particularly effective for long documents.

Document-type aware chunking — PDFs, markdown, HTML, and plain text all have different natural boundaries. A generic chunker treats them all the same and does all of them badly.

Embedding Model Selection Matters More Than the Vector Database

Teams spend weeks evaluating Pinecone vs Weaviate vs pgvector. They spend an afternoon picking an embedding model. It should be the opposite.

The embedding model determines the quality of your semantic search. The vector database is just storage with an index.

For most production use cases, the decision comes down to:

• text-embedding-3-large (OpenAI) — excellent quality, reasonable cost, easy to start with
• voyage-large-2 (Voyage AI) — often outperforms OpenAI on domain-specific retrieval
• BGE-M3 — open source, multilingual, runs on your own infrastructure

Always evaluate on your own data. Benchmark scores on MTEB don’t predict performance on your specific documents.

Retrieval Evaluation: The Missing Piece

Most teams deploy RAG and evaluate it by asking it questions and seeing if the answers seem right. This is not evaluation — it’s vibes.

Proper retrieval evaluation requires:

1. A golden dataset of question-document pairs where you know which chunks should be retrieved
2. Metrics: Recall@K (did the right chunks appear in the top K results?), MRR (where did the right chunk rank?)
3. A pipeline that runs these metrics automatically before any change to chunking, embedding, or retrieval goes to production

Building this takes a day. Not having it means you’re flying blind when you tune parameters.

Hybrid Search is Not Optional

Pure vector search misses exact keyword matches. Pure BM25 misses semantic similarity. In production, you need both.

The standard approach is to run both searches and combine scores using Reciprocal Rank Fusion (RRF). Most mature vector databases support this natively now — Weaviate, Qdrant, and Elasticsearch all have hybrid search built in.

The improvement in retrieval quality is consistently 15-25% in our experience. It’s the single highest-ROI change you can make to an existing RAG system.

The Infrastructure That Actually Matters

Caching — embedding queries is expensive. Cache embeddings for common queries. Cache retrieved chunks for repeated questions. A Redis cache in front of your vector database can cut costs by 60%+ on production workloads.

Observability — you need to know which queries are failing, which documents are never retrieved, and where the latency is. Log query embeddings, retrieved chunks, and final responses. Build dashboards on retrieval metrics, not just end-to-end response quality.

Async ingestion — document ingestion (parsing, chunking, embedding) should never happen in the request path. Use a queue and process asynchronously. Users should never wait for ingestion.

Re-ranking — after vector retrieval, run a cross-encoder re-ranker to re-score the top 20 candidates and return the top 5. Models like cross-encoder/ms-marco-MiniLM-L-6-v2 add 10-15% retrieval quality improvement with minimal latency overhead.

Conclusion

RAG in production is a systems engineering problem as much as an ML problem. The model is the easy part. The chunking, evaluation, hybrid retrieval, caching, and observability infrastructure is where production systems succeed or fail.

---

Building a RAG system and running into production challenges? Get in touch — we’ve shipped RAG at scale for enterprise clients and can accelerate your journey.