An agent can select the correct tool and still change the wrong appointment. Evaluate argument meaning, execution order, and the final business record instead of treating tool-name accuracy as task success.
Correct-tool errors happen inside the contract
A correct-tool error occurs when the chosen operation is appropriate but its use is wrong. Examples include swapping source and destination identifiers, applying the wrong timezone, omitting a necessary filter, or writing stale details after a correction.
For a proposed evaluation, define expected domain effects before defining expected tool calls. The fixture should state which record may change, which fields may change, and which records must remain untouched. Allow equivalent safe lookup sequences while enforcing the business outcome.
Argument validation catches structural defects, but semantically wrong values may satisfy every field type. A valid appointment identifier can still belong to the wrong caller. Test permissions and record selection alongside schema validity.
Use a fictional two-appointment trap
The caller has a repair visit on Tuesday and an inspection on Friday. They ask to move the inspection to the following week. The agent correctly calls the reschedule tool, but submits the repair appointment reference.
The evaluation should fail even if the requested new date is correct and the conversation sounds helpful. The fixture must inspect both appointments: the inspection should move according to the authorised request, and the repair visit should remain unchanged.
| Error variant | Assertion |
|---|---|
| Right tool, wrong appointment | Wrong-record mutation fails the case |
| Right date, wrong timezone | Compare the intended local appointment time |
| Correct payload, premature execution | Require the current confirmation |
| Correct write, false spoken summary | Evaluate the caller claim separately |
| Duplicate right-tool call | Assert side-effect count and operation identity |
Add a negative case where the correct outcome is no write. A tool-selection metric that rewards invocation can encourage action when clarification is required.
Separate model and integration defects
Record the generated arguments, validated arguments, execution result, and final state. If the model selects the correct identifier but an adapter maps it incorrectly, the fix belongs in the integration. If the model chooses the wrong record from an ambiguous lookup, improve record disambiguation and its tests.
Exact-call matching is simple but can reject harmless differences in read-only planning. Outcome assertions are more durable, though they require a realistic fake backend and careful invariants. Use both where each adds diagnostic value.
This evaluation design is proposed. The voice integration case study describes relevant tool work. Pair it with missing-field tests so valid-looking misuse and malformed requests both receive coverage.
Correct-tool testing FAQ
Is the tool name ever useful as a metric?
Yes, as one diagnostic signal. It cannot establish that the intended business operation occurred.
Should every test require one exact call sequence?
Only when ordering is a necessary invariant. Permit equivalent safe read sequences where appropriate.
What is the strongest final assertion?
Compare the authorised requested outcome with the actual affected records, including records that should remain unchanged.