Building production RAG: what changes between demo and reality

Retrieval-Augmented Generation is in production now across domains like legal research, technical support, and operational knowledge bases. Every domain teaches you something new about the gap between demo and production.

The naive approach

Most RAG tutorials follow this pattern:

1. Load documents
2. Split by fixed character count
3. Embed with OpenAI text-embedding-ada-002
4. Store in Pinecone
5. Retrieve top-3 chunks for every query
6. Stuff into GPT-4 prompt

This works for demos. It fails in production.

Chunking is everything

Fixed-size chunking destroys semantic boundaries. Legal clauses get split mid-sentence, code examples break across chunks, and tables become unreadable.

A more reliable approach is hierarchical chunking:

• Level 1: Document sections (H1/H2 boundaries)
• Level 2: Semantic paragraphs (NLTK sentence tokenization)
• Level 3: Sliding windows for overlap (20% overlap between adjacent chunks)

For structured data (JSON, tables, code), preserve structure integrity. A table row should never split across chunks.

Embedding models matter

text-embedding-3-large outperforms ada-002 significantly, but domain-specific fine-tuning is where the real gains are. On a domain with specialized vocabulary, fine-tuning a base embedding model on in-domain data can move retrieval accuracy from "usable" to "actually trustworthy."

The process:

1. Collect a few thousand query-document relevance pairs
2. Fine-tune with Multiple Negatives Ranking loss
3. Evaluate with MRR and NDCG@10
4. Deploy with fallback to the base model if the fine-tuned model drifts

Re-ranking is non-optional

Vector similarity finds roughly relevant chunks. A cross-encoder re-ranker finds precisely relevant chunks. A small re-ranker like ms-marco-MiniLM-L-6-v2 is fast enough for real-time use and consistently improves top-k precision.

A typical impact: irrelevant chunks in the top-3 drop by half or more once re-ranking is in the pipeline.

Query rewriting

Users do not write optimal queries. Real queries look like "that thing with the database." Add a rewrite step:

1. Extract entities and intent
2. Expand acronyms and synonyms
3. Rewrite as a declarative sentence
4. Generate 3 query variants
5. Retrieve for all variants and deduplicate

Recall typically rises meaningfully once user queries are normalized.

Evaluation framework

Production RAG needs continuous evaluation. A workable pipeline:

• Daily: Sample 100 queries, log retrieval results
• Weekly: Human annotators score relevance
• Monthly: Retrain re-ranker if drift detected
• Quarterly: Evaluate embedding model performance, consider fine-tuning

The hard truth

RAG is not a solved problem. Every deployment requires domain-specific tuning. The companies that treat it as infrastructure rather than a feature are the ones that succeed.