“The agent responds in one second” is hard to evaluate if nobody agrees when the clock starts or stops. A server metric can look fast while the caller still hears a long pause.
For conversational responsiveness, begin with the interval from the caller finishing a turn to the first audible response from the agent. Keep first-call startup and slow business actions as separate measurements.
Define the voice AI latency measurement boundary
An application may observe speech detection, transcription completion, model output, generated audio, and playback at different times. Those events describe different parts of the journey.
The LiveKit latency guide discusses several contributors and measurement boundaries. Do not add timings from overlapping pipeline stages and call the sum a caller's waiting time.
For a controlled test, document the recording method, phone or browser route, geographic setup, model configuration, and exact event definitions. An audio-based review and server timestamps answer complementary questions.
Separate ordinary replies from action-dependent replies
Create at least two test categories:
| Category | Example | What to inspect |
|---|---|---|
| Ordinary response | A question answered from approved business information | Turn completion to audible reply |
| External lookup | Appointment availability | Acknowledgement time and useful result time |
| First interaction | Opening a new session | Connection and initial audio separately |
An acknowledgement such as “Let me check” may reduce perceived silence. It does not reduce the time until the availability result arrives. Report both rather than counting the acknowledgement as task completion.
Use a repeatable set of turns
Test a short request, a longer explanation, a slowly spoken address, a mid-sentence pause, and a correction. Include the phone path if that is what customers will use. A quiet laptop demo does not reproduce every carrier and audio condition.
Review individual traces as well as a distribution. A median can hide occasional waits that make callers repeat themselves. Record the sample size and relevant percentiles without claiming more precision than the test supports.
If you change a model or turn-detection setting, repeat the same workload before deciding it improved the system. Also check interruption quality and task accuracy. A setting that answers sooner by cutting people off is not a clear improvement.
Optimise the measured bottleneck
If waiting is concentrated in a backend lookup, replacing the voice may do little. If the application waits too long to recognise the end of a turn, caching the CRM response will not address that delay.
Start with one trace, identify a dominant contributor, change one thing, and repeat the test. The slow-tool guide covers a common action-related delay.
Latency measurement questions
Is model response time the same as caller-perceived latency?
No. Audio processing, networking, playback, and application behaviour can also contribute.
Should we use one target for every conversation?
Different tasks have different constraints. Publish your boundary and workload instead of a context-free promise.
What evidence should an engineer provide?
A reproducible test and before-and-after traces, including accuracy checks. Bring a redacted example to a performance review.