A pleasant conversation can end with the wrong record, and an accurate workflow can feel frustrating. Score conversation quality and task completion separately so one does not hide the other.
Two scores answer two operational questions
Conversation quality describes the caller’s experience: clarity, turn handling, relevant questions, accurate explanations, and appropriate recovery. Task completion describes whether the authorised business outcome occurred with the right data and without forbidden effects.
In a proposed scorecard, use a concise reviewer rubric for conversational behaviours and backend assertions for task outcomes. Keep critical defects visible as named events rather than averaging them into a broad satisfaction number.
A task can be correctly deferred. If the integration is unavailable, an accurate explanation and an owned callback request may be the expected outcome. Define the scenario’s target before scoring so safe refusal is not automatically treated as failure.
Compare four fictional calls
One agent sounds natural and confirms an appointment, but no booking exists. Another asks one awkward clarification and creates the correct appointment after explicit confirmation. The first scores better on surface fluency but fails the task; the second may need conversational polish while meeting the business requirement.
A third call cannot book because the slot is unavailable and correctly offers alternatives. A fourth saves a request with the wrong address after a smooth conversation. These distinctions are lost when reviewers choose one overall rating.
| Conversation result | Business result | Interpretation |
|---|---|---|
| Clear and helpful | Correct outcome | Candidate for normal operation |
| Clear and helpful | Wrong or missing write | Serious workflow defect |
| Awkward but understandable | Correct outcome | Conversation improvement needed |
| Confusing | Wrong outcome | Investigate both layers |
Review timing too. A claim made before completion is unsupported even if the eventual record is correct. Task scoring should include when the system became entitled to make that claim.
Avoid a misleading combined average
A combined score can help sort examples, but it should not erase the underlying dimensions. Keep release-blocking conditions, such as unauthorised writes or false confirmations, outside any compensating average.
Human conversation ratings can vary by reviewer. Use examples, calibration, and a defined rubric. Backend assertions are more objective but can still encode the wrong business expectation, so operations should review those definitions as well.
This evaluation approach is proposed. The HVAC case study provides a concrete distinction between saved intake and confirmed appointments. Read false booking confirmation measurement for one critical outcome metric.
Evaluation score FAQ
Should fluency influence task completion?
No. Record whether the authorised outcome occurred independently of how polished the conversation sounded.
Can a safe fallback count as success?
Yes when that is the expected outcome for the scenario and the fallback is genuinely completed.
Should managers see only a single score?
Show separate dimensions and critical defect counts so operational decisions remain explainable.