A useful call summary preserves what matters without upgrading guesses into facts. Evaluate each consequential claim against the original conversation and tool outcomes, rather than scoring whether the summary sounds polished.
Summary accuracy has several dimensions
Call-summary evaluation checks factual support, omissions, speaker attribution, chronology, and action status. A summary can contain no invented sentences and still omit the unresolved issue that staff need to handle.
For a proposed review method, break the summary into atomic claims. Label each as supported, contradicted, unsupported, or uncertain because the source itself is unclear. Separately mark whether required information is missing, such as a correction, callback preference, or incomplete write.
The transcript is not perfect ground truth for every fact. Speech recognition may mishear a name, and a caller may state something unverified. Preserve those limitations. Tool evidence establishes whether an action completed; the conversation establishes what the caller said or confirmed.
Evaluate a fictional complaint summary
A caller reports a missed technician visit and requests a refund. The booking lookup is unavailable, and no refund tool runs. A summary saying “technician missed the appointment; refund approved” contains two unsupported upgrades.
A better summary states that the caller reports a missed visit, requests a refund, and needs staff review because appointment status was not verified. It should preserve the callback preference if the caller supplied one.
| Summary defect | Review question |
|---|---|
| Reported allegation presented as fact | Was the statement independently verified? |
| Request described as completed | Is there an authoritative execution result? |
| Old date retained after correction | Which value did the caller finally confirm? |
| Missing unresolved issue | Can staff identify the next action? |
| Wrong speaker attribution | Who actually made the statement? |
Have reviewers inspect source spans for consequential claims. A model-based evaluator can assist triage, but its judgment should not replace evidence when a disputed action affects a customer.
Measure usefulness separately from compression
A shorter summary is not necessarily better. Staff need the issue, relevant context, action status, and next owner. Evaluate whether someone can act without replaying the entire call, while retaining links to authorised source material.
Detailed review costs time and may expose sensitive information. Use access controls, minimised samples, and a clear review purpose. Keep ambiguous-source cases separate from definite summary errors so the metric points to the right fix.
This evaluation is proposed. The voice integration case study gives related context. See action evidence recording for linking summaries to execution facts.
Call summary FAQ
Is the transcript always correct?
No. Mark uncertainty when the audio or transcription is ambiguous rather than treating every token as verified truth.
Should omissions count as errors?
Yes when they remove information required for the defined staff workflow.
Can one accuracy percentage describe summary quality?
Use separate measures for unsupported claims, contradictions, required-field omissions, and practical staff usefulness.