Start with what the evaluation set must prove
Before collecting questions, define the jobs the RAG system must perform. An enterprise assistant may explain policies, compare product specifications, combine facts from several documents, or refuse when evidence or authorization is missing. These failures have different consequences: retrieving the wrong document, citing obsolete guidance, inventing a conclusion, and exposing restricted information should not collapse into one score for an answer that merely sounds reasonable.
We treat each case as a verifiable behavior specification. It should record the question, user context, permitted knowledge scope, expected evidence, required answer points, and whether refusal is appropriate. Capture the relevant corpus version, access conditions, and retrieval configuration as well. Without that context, a later result may be impossible to reproduce.
Build cases from risk and real usage
Seed the set from business workflows, domain-expert interviews, search or support logs, and threat modeling. Expert-written questions provide intent and authoritative expectations, but often omit abbreviations, typos, vague wording, and follow-up questions. Production-like queries supply that messiness, yet may preserve blind spots in the existing product. Use both sources and review them together.
- Core tasks: common policy, operational, and product questions with clear evidence.
- Long-tail tasks: less frequent but important questions requiring multi-section or multi-document reasoning.
- Hard negatives: documents with similar language but a different version, audience, product, or condition.
- Unanswerable cases: missing knowledge, ambiguous requests, or evidence that cannot support a conclusion.
- Authorization cases: the same question asked under different roles, departments, or data scopes.
The mix does not need to mirror traffic exactly. Keep a stable core set, a regression set focused on recent changes, and a sampled traffic set. This structure preserves comparability while giving high-risk scenarios enough visibility.
Label evidence, not only ideal answers
For every answerable case, annotate the smallest evidence needed to support the response: document identity, version, passage, and applicable conditions. If several sources are required, identify which ones are mandatory. This lets engineers test retrieval directly and separate a retrieval miss from a generation error where the model received the right material but misread it.
Represent the reference answer as required points and prohibited claims, rather than a paragraph the model must reproduce. For procedural or compliance content, also label ordering, exceptions, date sensitivity, and citation requirements. For unanswerable cases, record why refusal is expected and what a useful next step looks like, such as requesting missing context or directing the user to an approved system.
Evaluate retrieval, generation, and behavior separately
RAG is a pipeline, so one end-to-end grade rarely identifies what to fix. Evaluate against a fixed corpus snapshot and separate retrieval quality, evidence grounding, task completion, and system constraints. When an embedding model, chunking strategy, reranker, or prompt changes, this separation shows where the trade-off occurred.
- Retrieval: whether required evidence appears within the top k results, ranks sensibly, and outranks hard negatives.
- Grounding: whether every material claim is supported without dropping conditions, mixing versions, or filling gaps.
- Task completion: whether the response covers required points and follows citation, format, language, and actionability requirements.
- Safety: whether the system refuses unsupported requests, respects access boundaries, and resists instructions embedded in retrieved content.
Automated judges help scale coverage, but should not be the only authority. Retain deterministic checks and human review for critical cases, and audit whether the judge rewards verbosity, misses subtle conditions, or accepts unsupported statements. Release gates should be defined by risk slice rather than by an average score alone.
Maintain the set as a versioned engineering asset
Documents, permissions, models, and user behavior all change. A document revision, indexing change, or production incident should trigger an impact review: update the evidence, retain the old case as historical regression coverage, or retire it because the business rule no longer exists. Review and record each change so that passing a new release does not quietly become easier.
Store an owner, source, risk category, creation reason, corpus snapshot, and last review date for every case. Place new failures in a candidate pool, deduplicate them, and complete evidence labeling before promoting them into the core set. Results are most useful when reported by slice and failure cause. An evaluation set becomes operationally valuable when it reproduces real problems, identifies the failing layer, and prevents known regressions from reaching users.