Why Your RAG Pipeline Scores 94% in Testing and Fails in Production
How High Eval Scores Hide the "Lab Gap," Chunk Boundary Failures, and the Mirage of Synthetic Test Sets
Why Do RAG Systems Fail In Production Despite High Evaluation Scores?
RAG systems fail in production due to the "Lab Gap," where evaluation test sets are too similar to the training corpus. Standard evaluations often miss "Chunk Boundary" failures: where an answer spans multiple snippets, and "Query Distribution Shift," where real users use different terminology than the indexed documents. To fix this, teams must move beyond synthetic evals and implement strategies like Parent-Child Retrieval and continuous monitoring against real-world query failures.
A 94% evaluation score feels like a green light. In production, it becomes the number you cite when explaining why things went wrong.
RAG pipelines have a testing problem that is structural, not incidental. The way most teams evaluate RAG before deployment is fundamentally different from the conditions the pipeline encounters in production, and the gap between those two environments is where the failures live.
This is not a criticism of any specific RAG framework or evaluation tool. It is a description of how evaluation works, why it systematically overstates production performance, and what to do about it.
How Most RAG Evaluations Are Constructed
A standard RAG evaluation involves building a test set of question-answer pairs, running those questions through the pipeline, and measuring whether the retrieved context contains the answer and whether the generated response matches the expected output.
The metrics look rigorous: retrieval precision, retrieval recall, answer faithfulness, answer relevance. Tools like RAGAS, TruLens, and DeepEval provide automated scoring across these dimensions. A 94% score across a 200-question test set feels like strong validation.
The problem is in how the test set was built.
The Test Set Contamination Problem
Most evaluation test sets are built from the same document corpus used to build the production index. The questions are often generated by an LLM prompted to produce questions that the documents can answer. This creates a fundamental circularity: the test set is optimized for the documents, and the documents are optimized to answer the test set.
Real user queries do not work this way. Real users ask questions that span multiple documents in ways the test set generator did not anticipate. They use different terminology than the documents use. They ask questions that are adjacent to the indexed content but not directly answerable by it. They ask questions that require combining information from sources that would not appear in the same retrieved chunk under any reasonable chunking strategy.
A test set built from the corpus systematically underrepresents the queries where the pipeline will fail, because those queries do not look like questions the corpus can answer and the LLM-based test set generator tends not to produce them.
The Chunk Boundary Problem in Production
Chunking strategy is one of the most consequential decisions in RAG system design and one of the least visible failure points in evaluation. Most evaluation test sets use questions with answers that fall cleanly within a single chunk. The pipeline retrieves the right chunk, the answer is there, the evaluation records a success.
In production, the answers to many queries span chunk boundaries. A user asks about a process that is described across three paragraphs that were split into two chunks by a fixed-size chunking strategy. The retrieval system returns one chunk. The answer in the other chunk is never retrieved. The model either hallucinates an answer or says it cannot find the information.
This failure mode does not appear in standard evaluation because the test set generator tends to produce questions with self-contained answers. The chunk boundary problem is a production failure that evaluation almost never captures.
Query Distribution Shift
The test set used for evaluation represents a particular distribution of queries. That distribution reflects the assumptions of whoever built the test set about what users will ask.
Production query distributions drift from that assumption immediately. The first week of production use surfaces query patterns the test set did not include. Users ask follow-up questions that build on previous context. They ask meta-questions about the system’s capabilities. They ask questions that combine the indexed domain with external knowledge. They ask questions in languages other than the primary language of the corpus.
Each of these query types reveals failure modes that the evaluation score did not measure. A 94% score on the original test set tells you nothing about performance on queries that fall outside the test distribution, and in production, a significant fraction of queries will fall outside that distribution.
The Temporal Staleness Problem
Documents get updated. New documents get added. Old documents become outdated. Most RAG evaluations are point-in-time measurements against a static corpus. Production pipelines operate against a corpus that changes continuously.
When the corpus changes, retrieval quality can degrade in ways that are invisible until users start getting wrong answers. A document that was the authoritative source for a query at evaluation time may have been superseded by a newer document that was added to the corpus without updating the index properly. The retrieval system returns the old document. The answer is outdated. The evaluation score from three months ago does not reflect this.
What Production-Ready RAG Evaluation Actually Looks Like
Closing the gap between evaluation performance and production performance requires changing what you evaluate and when.
Evaluate on real user queries, not synthetic ones. As soon as you have production traffic, start logging queries that fail or receive low user satisfaction ratings. Build your evaluation set from these real failures. They represent the actual distribution your pipeline encounters.
Test chunk boundary spanning explicitly. Build a subset of your evaluation set with questions that require information from multiple chunks. Measure retrieval performance on these questions separately. The gap between single-chunk and multi-chunk query performance tells you whether your chunking strategy is a bottleneck.
Run continuous evaluation, not point-in-time evaluation. Set up automated evaluation that runs against a fixed test set on a schedule. Track the evaluation score over time as your corpus changes. A declining score on a fixed test set after a corpus update is a signal that the update degraded retrieval quality.
Evaluate retrieval and generation separately. A failure in generation is different from a failure in retrieval. If the right document was retrieved but the model generated a wrong answer from it, the fix is in the prompt or the model. If the right document was never retrieved, the fix is in the indexing or retrieval strategy. Aggregated end-to-end scores hide this distinction.
Implement structural fixes for chunk-based failures. If your evaluation reveals a high failure rate on queries spanning chunk boundaries, moving beyond basic fixed-size chunking is a requirement, not an option.
Techniques like Parent-Child Retrieval, where the system searches small and high-granularity chunks but returns the larger “parent” context to the LLM, ensure the model sees the full picture.
Alternatively, Contextual Compression can be used to dynamically extract and prioritize the most relevant snippets from multiple retrieved documents, effectively “stitching” the boundary gap before the generation phase begins.
The Number to Watch Instead of Overall Score
Overall evaluation score is the number that gets reported in sprint reviews. The number that actually predicts production failures is retrieval recall on hard queries: the percentage of queries where the correct document was retrieved when the query required spanning multiple chunks, using non-standard terminology, or combining information from disparate sources.
That number is harder to measure and worse-looking than the overall score. It is also the number that tells you whether your pipeline will hold up when real users start using it.
If You Read This Far, My Weekly AI Newsletter Is Probably For You.
Every Wednesday I send Pithy Cyborg | AI News Made Simple → 3 elite AI stories plus one prompt, no advertisers, no sponsors, no outside funding. One person. 10 to 20 hours of research. Straight to your inbox.
Always free. No paywalls. If it matters to you, a paid subscription ($5/month or $40/year) is what keeps it independent.
Subscribe free → Join Pithy Cyborg | AI News Made Simple for free.
Upgrade to paid → Become a paid subscriber. Support independent AI journalism.
If you’re not ready to subscribe, following on social helps more than you might think.
✖️ X/Twitter | 🦋 Bluesky | 💼 LinkedIn | ❓ Quora | 👽 Reddit
Thanks for reading.
Cordially yours,
Mike D (aka MrComputerScience)
Pithy Cyborg | AI News Made Simple
PithyCyborg.Substack.com