A multi-agent workflow passes only when the final business state matches the authorised request. Agreement among agents or a convincing final explanation cannot prove that the correct customer record changed exactly as intended.
Outcome tests should cross the agent boundary
Business-record evaluation compares starting state, authorised intent, expected mutations, and actual authoritative results. It should include records that must remain untouched and effects that must not occur.
For a proposed fixture, define the relevant customer, request, permissions, integration responses, and expected final state. Allow harmless differences in analysis or read-only tool order while enforcing consequential invariants.
Keep intermediate claims for diagnosis, but do not make agent self-evaluation the final authority. The execution service and target records provide different evidence from generated narrative.
Test a fictional wrong-appointment update
A caller has a repair appointment and an inspection. Several agents discuss the request correctly, but the execution stage submits the repair reference when the caller asked to move the inspection. Every agent may later repeat the same success summary.
The test must inspect both appointments. The inspection should reflect the authorised change, and the repair should remain untouched. A final answer containing the correct requested date is insufficient.
| Scenario | Final assertion |
|---|---|
| Routine authorised update | Correct resource and fields |
| Caller correction mid-workflow | Current confirmed revision used |
| Parallel proposals | One intended business effect |
| Tool outcome uncertain | No unsupported completion claim |
| Permission denied | No protected mutation or disclosure |
Add process restarts and out-of-order specialist responses. The same final record expectations should hold despite different timing, or the workflow should return an explicit safe unresolved state.
Separate record correctness from conversation quality
The caller still needs an understandable accurate explanation. Score that separately from the authoritative state. A correct update with misleading wording and a fluent conversation with a wrong update are different defects requiring different remedies.
Outcome fixtures take more setup than transcript-only grading. They are particularly valuable for consequential writes and duplicate effects. Use lightweight semantic review for low-impact exploratory tasks, while retaining strong record assertions where the business depends on correctness.
This evaluation harness is proposed. The voice integration case study describes related tool work without claiming multi-agent benchmark results. Pair it with correct-tool misuse testing.
Multi-agent outcome FAQ
Can agent consensus establish task success?
No. Verify the authorised effect in the system responsible for the business record.
Should the exact agent conversation be fixed?
Only where specific wording or ordering is required. Prefer durable business invariants for general evaluation.
What should a failure artifact include?
Keep starting state, expected outcome, configuration, relevant handoffs, operation history, and final records under appropriate data controls.