Human review is most useful when it targets uncertain judgments and consequential edge cases. Selecting only obvious successes creates a pleasant evaluation set that teaches little about release risk.
Reviewed examples need a selection purpose
A human-reviewed evaluation example is a scenario whose expected behaviour and observed outcome have been checked by qualified reviewers. Review can establish the business expectation, assess a generated conversation, or resolve an ambiguous label. State which role it serves.
For a proposed selection process, sample across intents, integration states, caller corrections, permission boundaries, and failure recovery. Include random ordinary cases alongside difficult or uncertain ones. Otherwise the set may be useful for debugging but misleading as an estimate of normal performance.
Keep a separate held-out subset for evaluating changes. If every reviewed example becomes prompt-development material, performance on that same set no longer provides a clean check of generalisation.
Select fictional booking examples deliberately
A product owner has 100 synthetic runs and review time for 20. They might allocate some review to ordinary random cases, some to all critical failures, and some to examples where automated checks disagree. The allocation is illustrative; its purpose should be documented.
Two reviewers independently assess whether “I have put that through” implies booking confirmation in context. If they disagree, an adjudicator resolves the label using the rubric and source evidence. Store the rationale so later reviewers do not repeat the same argument.
| Example type | Why review it? |
|---|---|
| Ordinary random success | Check for hidden errors in unflagged runs |
| Unsupported completion claim | Validate a consequential defect |
| Caller correction | Assess intent and chronology |
| Reviewer disagreement | Improve the label definition |
| Rare permission boundary | Confirm the expected safe behaviour |
Do not report the defect rate from an intentionally enriched failure sample as the overall production rate. Weighting or a separate representative sample is needed for that different question.
Review quality needs calibration
Provide concrete positive and negative examples for each rubric item. Reviewers should know the actual business commitment boundary and have access to appropriate tool evidence. A transcript alone may not reveal whether a request was saved.
Human review costs time and can expose sensitive data. Use fictional fixtures where possible, minimise real artifacts when necessary, and restrict access. Keep reviewer expertise visible: operational judgment and integration diagnosis may require different people.
This process is proposed. The voice platform case study offers related context. Read call-summary evaluation for a concrete claim-level review task.
Reviewed example FAQ
Should reviewers know which candidate produced a run?
Blind review can reduce preference bias when it is practical and the necessary diagnostic context remains available.
Are difficult examples enough for release evaluation?
They are valuable stress tests, but include representative cases for broader interpretation.
What happens when reviewers disagree?
Adjudicate against an explicit rubric and preserve the reasoning or mark the case genuinely ambiguous.