A failed evaluation does not automatically call for a longer prompt. Trace the first incorrect decision across speech, state, tools, and backend records before changing the component that happened to speak last.
Failure triage starts with the violated expectation
Evaluation triage identifies why an observed result differs from the scenario’s expected outcome. First confirm that the expectation reflects current business policy and that the fixture itself is valid. A stale expected date can look like an agent regression.
For a proposed triage record, capture the scenario version, candidate configuration, expected invariant, observed result, and relevant timeline. Find the earliest divergence that can explain the failure. Later conversational mistakes may be consequences rather than root causes.
Compare generated tool arguments with validated arguments and the backend result. This separates model selection errors from adapter mistakes, permission failures, and execution defects. Preserve the original failed artifact before rerunning.
Diagnose a fictional wrong-date booking
The conversation correctly confirms Thursday, but the saved appointment appears on Wednesday. The model’s tool arguments contain Thursday in the business timezone. An adapter converts the date through the server’s timezone and shifts it.
Adding “be careful with dates” to the prompt would not repair that transformation. The targeted fix belongs in the adapter and its timezone fixtures. After the fix, replay the original scenario and relevant neighbouring cases.
| Observation | Likely investigation boundary |
|---|---|
| Wrong value already in generated arguments | Intent or state handling |
| Correct arguments, wrong transformed payload | Adapter mapping |
| Correct payload, rejected operation | Domain rule or permission |
| Correct write, inaccurate final claim | Outcome interpretation |
| Expected result contradicts current policy | Fixture definition |
These are starting points, not automatic diagnoses. Confirm the mechanism with evidence rather than assigning blame from one log line.
Change one causal factor where possible
A prompt, model, schema, and retrieval update applied together can make a failed case pass without explaining why. Prefer a narrow fix and rerun the relevant regression suite. If several changes are necessary, preserve intermediate evidence where practical.
Extensive diagnosis can slow urgent recovery. In a live incident, reducing traffic or restoring a known version may come first. That operational response does not eliminate the need for later causal analysis and a durable test.
This method is proposed. The voice integration case study describes work across the relevant layers. Pair it with correct-tool misuse testing.
Evaluation triage FAQ
Should every failed run be repeated immediately?
Preserve its evidence first. Repetition can help establish variability but may not reproduce a race or external failure.
When is a prompt change appropriate?
When evidence shows the model’s interpretation or behaviour caused the defect and the change has a testable expected effect.
What closes the investigation?
A supported cause, a targeted remedy, and verification that the original failure and relevant regressions are addressed.