A retrieval-augmented generation system can fail in two very different ways. It may retrieve the wrong passages, leaving the model without the needed evidence, or it may retrieve the right material and still produce an unsupported or incomplete answer. A single “answer quality” score hides that distinction. RAG evaluation should measure the retrieval stage and the generated response separately.
Imagine an assistant for equipment manuals. When asked why a pump reports a particular warning, the system should retrieve the relevant model-specific section, use its conditions accurately and avoid inventing a repair step absent from the manual.
Build a query set from real work
Collect representative questions across common tasks, ambiguous wording, model numbers, outdated terminology and cases the corpus cannot answer. Remove or protect sensitive information and keep a held-out set that tuning does not repeatedly optimize against. For each query, record the user intent and the documents or passages a knowledgeable reviewer considers useful.
Include near-miss documents that share vocabulary but describe another product or procedure. Easy exact-match questions can make retrieval look healthy while masking the mistakes that matter operationally.
Evaluate retrieval before generation
Measure whether useful documents appear in the top results and whether irrelevant results dominate the limited context window. With relevance labels, ranking metrics can compare configurations; without them, structured human review remains valuable. Inspect failures by query type rather than averaging everything into one score.
Vary chunk size, overlap, filters and top-k deliberately. A larger context can improve recall while adding contradictory passages and cost. Enforce document-level access during evaluation so a configuration is never rewarded for retrieving content the test user should not see.
Grade grounding, relevance and completeness
Given the retrieved context, ask whether each factual claim is supported, whether the response addresses the question and whether it omits a critical condition. Microsoft's RAG evaluator guidance distinguishes retrieval process evaluation from response groundedness, relevance and completeness. That separation helps a team choose the correct fix.
Automated judges can scale review but need calibration against human labels. Record model and evaluator versions, prompts, thresholds and disagreement examples. Do not present a judge score as objective truth; use it as one signal with reproducible examples.
Connect evaluation to the operating workflow
Measure abstention and escalation. When the corpus lacks evidence, a useful system should say what is missing or route the question, not improvise. Track citations that open the exact supporting section. Review latency and cost beside quality because a configuration that cannot meet the product's operating constraints is not ready.
Run the suite when the corpus, embedding model, retrieval configuration or generation model changes. A strong RAG evaluation tells engineers where quality degraded and gives product owners concrete examples. The outcome is not a universal score; it is evidence that the system retrieves the right material and uses it responsibly for the questions it claims to support.
For related delivery support, explore 4Minds Technology's AI and automation services and custom software development. A useful implementation should leave the operating rule, evidence and ownership clear enough for the next team member to maintain. Record the final decision beside the system so future changes can be reviewed against the original purpose.



