Troubleshooting RAG Retrieval Quality: Find the Evidence Behind Poor Answers

Introduction

A RAG system can write a fluent answer and still miss the facts a user needs. Poor results may start with retrieval, generation, or both, so a polished response alone won’t tell you where the fault lies.

Start by inspecting the passages returned for real user queries before changing prompts or models. The evidence shows whether relevant content is missing, ranked too low, or reaching the model without being used.

Retrieval quality can be measured, but no single score explains every failure. A repeatable test set with real queries, relevant documents, and traceable results gives you a better path to targeted fixes.

Find the Retrieval Failure Before Changing the Model

First locate where the answer went wrong. This keeps you from tuning the language model when the real problem is an empty result, a poor match, or a bad filter.

Trace a failed answer to its retrieved passages

For each failed query, review the original text and any rewritten version. Then check document IDs, chunk text, scores, filters, and the final context sent to the model. If the evidence needed to answer never appeared in the results, focus on retrieval rather than generation.

Separate retrieval failures from generation failures

A relevant passage may be missing, buried below the retrieval cutoff, or present in the context but ignored or misread. These are different failure types and need different fixes. Answer quality or faithfulness scores alone can’t show whether retrieval found the right evidence.

Build a test set from real queries

Save anonymized production queries and label the evidence each one needs, along with relevant passages and failure type. Include common questions, hard-to-match wording, and high-impact requests. A test set made only from hand-picked successes can hide the gaps users face.

Measure Retrieval Quality With Metrics That Match the Failure

Choose metrics based on what you’re trying to diagnose: whether useful evidence appears, how high it ranks, and whether it stays visible to the model. Review results by query type as well as across the full test set.

Track recall and precision at the right cutoff

Recall@k checks whether relevant evidence appears in the first k results. Precision@k measures how much of that result set is relevant. Set k to match the number of passages your model can use; results beyond that limit may never affect its answer.

Use ranking metrics to spot misplaced evidence

Mean Reciprocal Rank (MRR) rewards a relevant result near the top by measuring the position of the first relevant passage. Normalized Discounted Cumulative Gain (nDCG) accounts for ranked results with different levels of relevance. Both can reveal ranking problems that a simple “did we find it?” check misses.

Keep retrieval tests separate from answer checks

Maintain relevance judgments for retrieval alongside checks for answer support and relevance. BEIR (Thakur et al., 2021) offers a benchmark for information retrieval, while MTEB (Muennighoff et al., 2023) compares text embedding models across tasks. RAGAS (Es et al., 2023) covers multiple parts of RAG evaluation. Use these as references, not replacements for tests based on your own data.

Fix Corpus and Chunking Problems That Hide Relevant Evidence

Retrieval can’t find information that never made it into the index in a usable form. Check ingestion and text processing before changing the search method.

Check for missing, stale, or damaged content

Confirm that the right sources loaded, parsed correctly, and refreshed on schedule. Look for duplicate or empty records, then compare indexed text with the source. Tables, headings, PDFs, and structured data can lose key details during extraction.

Tune chunk size and boundaries around meaning

Chunks that are too small may split a definition from its explanation. Large chunks can contain useful facts but also enough unrelated text to weaken the match. Test boundaries that keep procedures, definitions, and section links intact; add overlap only when evaluation shows it helps.

Preserve metadata and document structure

Keep fields such as title, section, date, source, and access permissions with each passage. Metadata can support filters, source citations, and freshness checks. It also makes failed results easier to trace back to a document and the rules used to retrieve it.

Improve Matching When Queries and Documents Use Different Language

A useful passage can sit in the index and still fail to match the way a person asks a question. Check for gaps in wording, context, and exact terms before replacing your embedding model.

Test embedding coverage against real queries

Compare real queries with their known relevant passages using your current embedding model. Inspect misses involving abbreviations, domain terms, other languages, or specialist wording. Test candidate models on your labeled queries; MTEB gives broader context, but its results can’t prove a model fits your domain.

Add query rewriting only for proven gaps

Rewriting can clarify a follow-up such as “Does it work with that version?” or expand an acronym found in internal documents. Log both the original and rewritten query, then check that the new wording keeps the user’s intent. A rewrite that adds an assumption may retrieve the wrong evidence.

Combine semantic and lexical search for exact terms

Semantic search helps match related ideas, while keyword search can catch product names, IDs, error codes, quoted text, and rare terms. Hybrid retrieval combines these signals. Compare methods on the same test set and inspect the passages they return, rather than choosing by score alone.

Correct Ranking, Filtering, and Retrieval Configuration

When relevant content is indexed, check whether search settings let it reach the model. Filters, candidate depth, and reranking can all change which passages make the final context.

Inspect filters, permissions, and freshness rules

Review metadata filters and date or version limits against the query’s needs. Check that access controls allow the right users to see the right sources. Logs can reveal an empty candidate set or a filter that narrowed results too far.

Tune candidate depth and reranking with evidence

Retrieving more candidates may improve recall, but can also add noise and latency. A reranker can improve the order of candidates it receives; it can’t recover a document that first-stage search never returned. Compare both rankings and track quality and response time together.

Make retrieval decisions observable

Log query rewrites, filters, candidate IDs and scores, reranker output, and the final model context. Version the index, embedding model, and retrieval settings. These records let you reproduce a regression and identify which change caused it.

Turn Retrieval Fixes Into a Reliable Evaluation Loop

A retrieval fix needs to work beyond the query that exposed the problem. Rerun a stable test set after each change and keep checking results as sources and user language shift.

Validate changes against real queries

After changing chunking, embeddings, query processing, filters, or ranking, rerun the same evaluation set. Review aggregate scores and individual regressions, especially for high-risk questions. Change one part at a time so you can tell what helped.

Monitor retrieval as the corpus changes

Track retrieval metrics and sample query traces over time. Refresh labels when source content or user wording changes, and investigate shifts in empty results, relevance, latency, or answer support. A passing test today doesn’t guarantee tomorrow’s content will retrieve well.

Conclusion

Troubleshooting RAG retrieval quality starts with the evidence: inspect the passages, classify the failure, and measure the behavior that needs to improve. Fix the source, matching, filtering, or ranking issue before changing generation settings. Run your real-query test set after each change, then keep monitoring it as the corpus grows.