An AI phone test can sound correct while saving the wrong address. If your evaluator only reads the transcript, it may approve a conversation that failed the business.
For workflows that change data, test the final records as well as the words the agent used.
What is a saved-record assertion?
It is a check that the system state matches the expected result after a controlled test. The expected result might be one appointment, a corrected service address, a pending callback task, or no write at all.
Use fictional fixtures with known starting conditions. Give each run a reference so you can find its records without searching real customer data.
Write expectations before recording the call
Consider a fictional caller who changes a preferred day and then discovers the calendar is unavailable:
| Field or outcome | Expected state |
|---|---|
| Requested day | Latest confirmed preference |
| Appointment | None confirmed |
| Follow-up task | Created with an owner |
| Caller-facing wording | Availability remains unconfirmed |
This test should fail if the receptionist says the visit is booked, even when the fallback task was created correctly. It should also fail if the conversation sounds careful but the task has no usable callback number.
Separate assertions by system boundary
Check the conversation, the tool input, the operation log, and the destination record. That separation narrows the diagnosis.
If the transcript contains the corrected address but the tool input does not, inspect the conversation-to-action boundary. If the tool input is correct but the destination record is wrong, inspect mapping, validation, and provider behaviour.
Testing external APIs through a safe environment is important. Twilio's test-credential documentation illustrates that providers may offer specific testing mechanisms, but those do not reproduce every live feature. Verify what your chosen test path actually exercises.
Include outcomes where the correct result is no action
A caller may withdraw a request, provide incomplete information, or lack authority for the requested change. A test suite that always rewards a created record can encourage the wrong behaviour.
Include a repeated delivery, timeout after a possible write, caller correction, failed transfer, and cancellation. Wait for the workflow's agreed settlement point before asserting final state. Asynchronous processing can make an immediate check misleading.
For an uncertain outcome, expect a visible unresolved state and a reconciliation path. Do not quietly convert uncertainty into a pass.
Keep the suite small enough to rerun
Start with the highest-cost failures, then add cases discovered during review. A short suite that runs after every relevant change is more useful than a large spreadsheet nobody executes.
Keep provider-specific checks separate from portable business expectations. That makes it easier to change implementation without rewriting what a successful customer outcome means. The duplicate-booking article provides another useful failure category.
Record-testing questions
Is an HTTP success response enough?
No. Inspect the intended record and its business state.
Must every test call real external systems?
No. Use isolated tests for logic and a smaller integration suite for supported provider behaviour.
What should a client see at sign-off?
The scenario, expected outcome, actual record, and unresolved limitations. Use that format in a delivery review.