A release-blocking check should protect a specific business invariant. “The calls sounded good” is too subjective to stop a deployment that creates duplicates, loses requests, or confirms work that never completed.
Release gates need clear failure criteria
A release gate is an explicit condition that a candidate must satisfy before receiving traffic. For voice workflows, useful critical invariants include authorised writes only, no unsupported completion claims, no duplicate side effects, and recoverable handoff failures.
In a proposed release process, classify checks by consequence. Some defects block any rollout. Others require review or can enter a monitored pilot. Assign an owner for each decision and preserve the failed fixture and configuration.
Do not choose thresholds after seeing the candidate’s results. A predeclared rule makes it harder to excuse a serious failure because a new voice or shorter prompt feels more appealing.
Build a fictional booking gate
A team prepares a prompt update for appointment intake. Its blocking suite contains confirmation refusal, stale-slot rejection, duplicate retry, cross-account access denial, and lost-write-response cases. Each test inspects final records and caller-facing claims.
The candidate performs better on conversational brevity but creates a second appointment after a simulated response loss. The release should remain blocked under the duplicate-write rule. A higher average quality score cannot compensate for that specific invariant violation.
| Check | Example blocking condition |
|---|---|
| Confirmation boundary | Write occurs after caller declines |
| Permission boundary | Unrelated record is modified |
| Idempotency | One intent creates two appointments |
| Outcome honesty | Failed write receives a success claim |
| Handoff recovery | Caller disconnects when staff rejects |
Run the critical suite against the deployable artifact and configuration, not an earlier local variant. Record the prompt, model, tool-contract, and knowledge versions so evidence maps to the actual candidate.
Gates complement a monitored pilot
A passed synthetic suite proves only the tested expectations under the tested conditions. Keep rollback or traffic-reduction procedures and monitor the same failure categories during the pilot. Human review can reveal ambiguity the automated checks did not encode.
Overly broad gates can slow harmless changes and encourage workarounds. Keep each blocking rule tied to a material consequence and a reproducible test. When a gate is noisy, repair the test or definition rather than normalising overrides.
This process is proposed. The voice platform case study describes related engineering scope. Review prompt regression tests for comparing a candidate with the current release.
Release gate FAQ
Should every conversational flaw block release?
No. Classify severity according to operational consequence and the deployment’s requirements.
Can a good average score override a critical failure?
Not under a rule that defines the failure as blocking. Fix or explicitly redesign the requirement before release.
What should an approved exception include?
A concrete risk explanation, accountable owner, bounded rollout conditions, and evidence that the remaining exposure is understood.