A higher average evaluation score does not justify releasing a knowledge assistant that now exposes a restricted document. Release gates should protect specific product boundaries before considering aggregate improvements.
For a company assistant, the relevant boundaries usually include authorised retrieval, supported decisions, useful abstention, and recoverable operations. Decide which failures block release before looking at a candidate's results.
Divide release criteria by consequence
Create a critical set for permission leakage, unsupported high-impact instructions, and failures that prevent core work. Create a broader quality set for common questions, source citations, version handling, and response usability. Record the corpus revision and test identities with both sets.
Use explicit pass conditions. “Good citations” is too vague. “Every material policy claim in these reviewed cases links to the supporting revision and passage” gives a reviewer something concrete to verify. The test remains a sample, so it should not be described as proof that all future citations will be correct.
Review a fictional release candidate
Imagine a candidate answers more routine support questions correctly but fails a department-transfer test: a user whose access was removed still receives a protected answer from cache. Under an access-boundary gate, that candidate remains blocked even if the overall score improves.
Assign the failure to an owner, preserve the trace, and require a repeat of the failed case plus adjacent cases. Those adjacent checks should include a still-authorised user, a fresh conversation, an existing conversation, and the citation destination. A broad model retest alone may miss the cache defect.
For a noncritical wording regression, the decision may differ. If the answer is supported and actionable but awkward, the team can accept a documented follow-up. Keep that exception narrow and attach an owner and expiry rather than turning “known issue” into permanent permission to ship.
Add an operational gate to the evaluation gate
Confirm that staff can identify the deployed version, report a failed answer with its trace, disable affected sources, and return to the prior configuration. A rollback that depends on a departing developer's laptop is not an operational plan.
Offline cases cannot cover every live permission state or source outage. A limited rollout with monitored failure categories can reveal additional problems, provided users know the assistant's scope and the team can respond. Choose rollout size from the impact and support capacity, not a universal percentage.
Automated judging helps prioritise review, while human checks remain useful for business consequences and ambiguous sources. The official RAG evaluation reference describes distinct evaluation dimensions. Your release policy must still decide which outcomes are acceptable.
See document integration planning and the Nucleus project account for related context.
Questions about release gates
Should every failure block release?
No. Classify consequences in advance. Access violations and misleading critical decisions deserve different treatment from minor formatting defects.
Can the candidate use a new corpus snapshot?
Yes, but record that change and separate it from the assistant-version comparison. Otherwise, easier source material can look like better engineering.
What should a release review contain?
Include blocking-case outcomes, accepted exceptions, owners, source versions, and rollback evidence. Bring that packet to a technical review before expanding access.