If the questions, documents, and scoring rules all change between runs, a better score cannot tell you whether the assistant improved. A fixed evaluation set creates a controlled comparison, provided you preserve the conditions around it.
For product owners, the useful unit is a paired result: how the old and new versions handled the same task with the same authorised evidence. Aggregate scores come after that comparison.
Freeze more than the question list
Record the case revision, source snapshot, reader identity, expected answer conditions, rubric, model configuration, retrieval settings, and evaluation tool version. Some hosted dependencies can still vary. Document what you cannot freeze rather than describing the run as perfectly reproducible.
Keep tuning cases separate from held-out cases. Once the team repeatedly optimises against a question, its result becomes less informative about unfamiliar production requests. Preserve the case for regression testing and add fresh reviewed cases for broader evaluation.
Compare a fictional policy assistant change
Suppose version A often misses footnotes and version B changes table parsing. Run both against identical policy questions, including a table-based allowance, a historical policy, an inaccessible source, and a genuinely unanswered question.
For each pair, record required facts covered, unsupported claims, citation correctness, permission behaviour, and task completion. The table case may improve while the historical-policy case regresses because the new parser loses revision metadata.
Do not describe that candidate simply as “better at retrieval.” Report the gain and regression separately, preserve both traces, and determine whether the regression crosses a release boundary. A critical access failure should not disappear inside an average of many easy questions.
Where generation varies, run each version more than once under the same plan. Report observed variation and sample size. A single lucky response cannot establish that a recurring defect has been fixed.
Keep benchmark changes as their own versioned work
When a policy changes, update the evaluation case with a new source revision and explain why its expected answer changed. Rerun both assistant versions on that new snapshot if you need a fair comparison. Do not silently replace the reference answer and compare the new score with an old report.
Fixed sets can become stale. Add new failures, emerging workflows, and changed permissions through a reviewed benchmark revision. Keep prior reports so the team can distinguish product improvement from changing coverage.
Automated metrics can make the paired review easier to sort. The official RAG evaluator reference describes multiple evaluation dimensions; your review still needs source-specific expectations.
The citation guide and Nucleus project account provide related context.
Questions about version comparisons
How many cases are enough?
There is no universal count. Cover the important workflows and failure categories, disclose the sample, and expand where uncertainty remains.
Can live documents be used?
Yes, for a separate freshness check. Use recorded snapshots for controlled version comparison so source changes do not obscure the result.
What should the decision report show first?
Lead with critical regressions, then task-level gains, remaining failures, and limitations. Bring the paired examples to a knowledge assistant review instead of sharing only an average score.