A fictional customer scenario becomes a useful test when it specifies an expected business outcome. A persona and a conversational script alone do not tell you whether the assistant performed the task correctly.
Fictional scenarios need operational detail
A voice AI test set is a collection of controlled interactions with known expectations. Fictional data avoids exposing real customer records and allows you to create edge cases deliberately. It does not establish how frequently those cases occur in production.
For a proposed fixture format, include the caller’s goal, relevant starting records, permitted information, integration behaviour, expected mutations, forbidden effects, and acceptable recovery. Write the expected outcome before asking an agent to run the conversation.
Use recognisably fictional contact details and isolated test tenants. Do not route messages or calls to invented numbers that may belong to real people. The harness should intercept external effects regardless of the values in the fixture.
Build a fictional repair scenario family
Start with a caller requesting service for a noisy furnace. The complete case includes a covered address, valid callback information, and explicit confirmation. The expected result is one service request with the confirmed issue and contact details.
Create related cases by changing one consequential condition: the address is outside the service area, confirmation is declined, or the write service times out. Each case should have its own expected result. Reusing the happy-path expectation would conceal the branch you intended to test.
| Variation | Expected result |
|---|---|
| Covered address, confirmed intake | One correctly populated request |
| Unsupported location | No unsupported service promise |
| Caller declines saving | No request mutation |
| Caller corrects callback number | Only the latest confirmed number saved |
| Save outcome unknown | Reconciliation, no false confirmation |
Have an operations reviewer check whether the scenario and recovery are plausible. An engineer can create a technically valid test that contradicts how the office actually handles requests.
Balance coverage and realism
A large set of paraphrases can make the dataset look broad while exercising the same branch repeatedly. Track coverage by intent, missing information, correction, permission boundary, integration state, and final outcome. Add linguistic variation after those behaviours are represented.
Synthetic scenarios are easiest to maintain when each has a clear purpose and owner. Retire obsolete policy expectations, but keep a record of why they changed. Otherwise an updated fixture can silently erase evidence of a regression.
This dataset design is proposed. The HVAC demo case study offers a concrete request workflow and explicitly uses fictional demo records. See human-reviewed evaluation examples for review selection.
Fictional test-set FAQ
Can synthetic cases replace pilot calls?
No. They establish controlled coverage; a pilot tests real-world behaviour and frequency.
Should every case use a different persona?
Only when the distinction affects the behaviour being tested. Operational variation matters more than decorative biography.
How many cases are enough?
Start with coverage of critical branches and invariants, then expand from observed gaps rather than targeting an arbitrary dataset size.