Test retrieval separately by freezing a corpus, writing representative questions, marking the passages that support their answers, and inspecting what the retriever returns. Then check whether those passages actually reach the answering model. A model change cannot recover a source that the pipeline never supplied, although it can change how the available evidence is interpreted.
A wrong answer does not identify the failing component. The source may be missing, the query may miss it, a filter may exclude it, the ranker may bury it, or context assembly may discard it. Start by locating that break in the evidence path.
Define relevance before inspecting the ranking
For a synthetic export-policy corpus, create four identifiable passages: the standard retention rule, a trial exception, the deletion procedure, and an unrelated page. Give each a stable document ID, revision, and passage location.
Now write questions and the evidence each needs. “How long are trial exports retained?” requires both the general rule and its exception for this deliberately constructed evaluation. “How do I delete an export?” needs the deletion procedure. “Which country stores my exports?” has no supporting passage in this corpus.
That last case matters. A retrieval system often returns something even when it cannot answer the question. A plausible-looking hit is not evidence that the answer exists.
Microsoft's RAG evaluator documentation separates process evaluation of retrieved documents from evaluation of generated responses. It also distinguishes ground-truth relevance labels from model-judged relevance. Use those distinctions even if your test harness has no Microsoft dependency.
Inspect ranked evidence, not only a final score
A minimal case record contains the question, applicable corpus snapshot, required evidence IDs, returned IDs in order, filters, and retrieval configuration. Keep the exact text handed to generation as another artifact. The retriever's logs and the final prompt can differ.
Consider these constructed top-two rankings:
| Question | Relevant IDs | Returned IDs | Interpretation |
|---|---|---|---|
| Trial retention | standard, exception |
standard, noise |
The decisive exception is missing |
| Deletion procedure | delete |
delete, standard |
Required evidence is present, with one irrelevant hit |
| Data residency | None in this corpus | standard, delete |
Neither returned passage supports an answer |
For this fixture, precision at two is the number of relevant returned IDs divided by two. Recall at two is the number of relevant returned IDs divided by the number of known relevant IDs. The Azure information-retrieval guide discusses these retrieval measures alongside ranking metrics.
The first case has precision 0.5 and recall 0.5. The second has precision 0.5 and recall 1.0. The third has precision 0 and undefined recall because its relevant set is empty. We represent that recall as null and report the no-answer case separately, rather than manufacture a perfect score for returning nothing useful.
A local Python fixture calculated and checked these values. It graded handwritten rankings; it did not run a search engine or model. Its purpose is to expose the denominator and missing-evidence decisions before a real evaluation inherits them.
Choose the evidence unit deliberately
Chunk-level labels can overcount duplicated evidence. If ten overlapping chunks all repeat one rule, retrieving five of them may look productive while still omitting the exception. Group equivalent passages when the question is whether a fact was recovered, and retain chunk identities when debugging the splitter or ranking.
Conversely, a single “relevant document” label may be too generous for a long manual. Finding the right manual does not establish that the needed page reached generation. Use passage or evidence-group labels at the point where they match the system's output.
A relevance label also has a scope: user, permissions, product version, jurisdiction, or date may change which passage is eligible. Evaluate within the requesting identity's allowed corpus. A result containing the perfect answer from an inaccessible document is an authorization failure, not a retrieval success.
Trace one failed answer through the pipeline
For the trial question, inspect the source first. If the exception is absent from the indexed revision, repair ingestion or freshness. If it exists in storage but no chunk retains its scope, inspect extraction and chunking. If a complete chunk exists but ranks below the cutoff, examine query terms, filters, retrieval method, and reranking.
If the chunk is in the returned results but absent from the assembled context, inspect deduplication, ordering, and truncation. Only when it is present and legible in the final input should the investigation focus primarily on the answering instruction, model behavior, or output handling.
This sequence is not a promise that one component caused the failure. Multiple defects can coexist. It makes each change answer a narrower hypothesis and prevents a model upgrade from concealing a broken evidence pipeline.
Evaluating a coding agent on your repository covers the wider task-level evaluation. Here the unit under test is the retriever and the context it supplies, so a correct final answer from prior model knowledge does not rescue a missing-evidence case.
Compare changes without tuning to the answer key
Keep a development query set for diagnosing failures and a separate held-out set for the final comparison. Change one meaningful retrieval choice at a time, or explicitly label a combined intervention. Preserve query rewrites, index versions, filters, ranking parameters, and the context budget.
Report per-question outcomes before aggregating. A higher average can hide a regression in exception-heavy or no-answer cases. If an LLM judges relevance, inspect disagreements against human labels and keep the judge configuration separate from the retriever configuration.
After retrieval improves, evaluate generated answers again. Evidence coverage is necessary for grounded corpus-specific answers, but it does not prove that the model interpreted the evidence correctly. The useful result is a traceable explanation of what failed and which change corrected it.
Research and metric-fixture date: October 28, 2026. No retrieval-quality claim about a deployed service, model comparison, or paid evaluation is made.
