A RAG system can return a wrong answer for at least two independent reasons: it retrieved the wrong evidence, or it generated a bad answer from good evidence. One end-to-end score cannot tell you which component to fix.
The useful evaluation model is therefore a pipeline:
question → retrieval → context → generation → final answer↑ ↑retrieval metrics answer metrics
This guide turns that pipeline into a test plan. If you are still deciding whether retrieval is the right architecture, start with RAG vs fine-tuning vs long context.
Evaluate four different relationships
The cleanest RAG scorecard compares four pairs:
| Evaluation | Comparison | Question answered |
|---|---|---|
| Retrieval relevance | retrieved documents vs user question | Did we fetch material about the request? |
| Context coverage | retrieved documents vs reference evidence | Did we fetch enough evidence to answer? |
| Groundedness | answer vs retrieved documents | Does the answer stay within the evidence? |
| Answer correctness | answer vs reference answer | Is the final answer actually right? |
Add answer relevance—answer versus user question—when systems tend to produce correct but unhelpful tangents.
These dimensions should remain separate. A grounded answer can still be wrong when the retrieved document is outdated. A correct answer can be ungrounded when the model relied on prior knowledge rather than the supplied evidence.
Build a dataset that exposes retrieval failures
Start with real questions, not a list generated entirely from the same documents the system indexes. Document-derived questions tend to use the source vocabulary and make retrieval look easier than it is.
For each test case, capture:
{"question": "Can contractors access the production analytics workspace?","reference_answer": "Only after security approval and time-limited access is granted.","relevant_document_ids": ["access-policy-v4"],"relevant_passages": ["Contractor access requires..."],"category": "permissions","difficulty": "multi_condition"}
Include the cases that polished demos omit:
- questions using customer vocabulary instead of document vocabulary;
- answers split across multiple documents;
- outdated documents that should lose to newer versions;
- questions with no answer in the corpus;
- ambiguous questions that require clarification;
- authorization-sensitive questions where the user must not retrieve certain evidence.
Twenty carefully reviewed cases are more useful than 2,000 synthetic cases nobody has inspected. Expand the dataset from production failures after the first release.
Retrieval metrics that reveal different problems
Recall at K
Recall@K asks whether at least one required document or passage appears in the first K results. It is the right first metric when missing evidence is the dominant failure.
Recall@K = relevant items retrieved in top K / relevant items available
High recall with poor answer quality points toward generation, context ordering, or noisy retrieval. Low recall means prompt tuning cannot rescue the system consistently.
Precision at K
Precision@K measures how much of the retrieved set is relevant. Low precision wastes context and can distract the generator.
Precision@K = relevant items retrieved in top K / K
Do not maximize precision blindly. Returning one relevant passage produces perfect precision and may omit evidence required for a multi-document answer.
Mean reciprocal rank
Reciprocal rank rewards placing the first relevant result near the top. It matters when the model or UI weights early results heavily.
RR = 1 / rank of first relevant result
Average reciprocal rank across the dataset to get MRR. Use it alongside recall rather than as a replacement.
Context coverage
Document relevance is not enough for questions requiring several facts. Context coverage scores how many reference claims have supporting evidence in the retrieved set.
For a reference answer with four required claims, retrieving evidence for three yields 75% coverage—even when every returned passage is individually relevant.
Answer metrics that should not be collapsed
Groundedness
Break the answer into factual claims, then determine whether each claim is supported by the retrieved context.
Groundedness = supported answer claims / factual answer claims
An LLM judge can scale this check, but calibrate it against human labels. Require the judge to return the claim, supporting passage, and verdict so reviewers can audit disagreements.
Correctness
Compare the generated answer with the reviewed reference answer. Exact string matching works for IDs and structured values. Semantic grading works better for prose, provided the rubric defines which claims are required and which errors are critical.
Answer relevance
Check whether the answer directly resolves the user's request. This catches evasive responses, unnecessary background, and answers to a nearby question.
Refusal quality
For unanswerable questions, the desired output may be a refusal, a clarification request, or an explicit statement that the corpus lacks evidence. Score that behavior separately. A system that invents an answer should not receive credit merely because the response sounds relevant.
Diagnose with a metric matrix
| Retrieval | Groundedness | Correctness | Likely diagnosis |
|---|---|---|---|
| Low | Low | Low | Retrieval is the first bottleneck |
| High | Low | Low | Generator ignores or misuses evidence |
| High | High | Low | Evidence is incomplete, stale, or reference is wrong |
| High | High | High | Healthy path; inspect latency and cost next |
| Low | High | High | Model may be answering from prior knowledge |
This matrix prevents the common response to every failure: changing the prompt.
Run offline and online evaluation differently
Offline evaluation uses a curated dataset before release. Run it when changing:
- embedding models;
- chunking or metadata filters;
- top-K and reranking;
- prompt or model versions;
- document parsers;
- index contents.
Online evaluation samples production traces. It is best for groundedness, answer relevance, citation validity, user corrections, and emerging query categories. Production feedback should create new offline cases, closing the loop described in the LLM observability guide.
A release gate that teams can operate
Define thresholds per category, not just globally. A 92% average can hide a serious failure in permissions or billing questions.
Example release policy:
- No critical authorization case may regress.- Recall@5 must not fall by more than 2 percentage points.- Groundedness must remain above 95% on policy questions.- Unanswerable-question refusal accuracy must remain above 90%.- P95 latency and cost per successful answer must stay within budget.
Always inspect the changed examples. A score tells you that behavior moved; the examples tell you whether the evaluator or the application is wrong.
RAG evaluation checklist
- Retrieval and generation receive separate scores.
- Dataset cases include relevant document and passage identifiers.
- No-answer, stale-content, and multi-document cases are represented.
- Automated graders are calibrated against human labels.
- Results are segmented by question category and difficulty.
- Index, embedding, chunking, prompt, and model versions are recorded.
- Production failures become regression cases.
- Release thresholds include quality, latency, and cost.
