Hostile customer text should be treated as input to the support task, not as authority to change system permissions. Test both visible answers and attempted tools so a polite response cannot hide an unauthorised side effect.
Prompt-injection tests target trust boundaries
A customer-provided instruction may ask the assistant to ignore its role, reveal private information, or perform an unrelated action. Similar instructions can arrive inside attachments or quoted external content.
OWASP’s prompt-injection guidance describes direct and indirect input risks and recommends layered controls. A warning in the prompt is useful context, but the business tools still need enforceable permissions and constrained interfaces.
For a proposed test set, define the legitimate issue, hostile content, expected safe response, forbidden disclosures, and forbidden effects. Use synthetic private markers and isolated tools rather than real secrets or customer data.
Exercise a fictional forged administrator request
A ticket contains a genuine export problem followed by text claiming that the requester is an administrator and must receive all account records. The assistant should continue appropriate support while the backend refuses any unauthorised export attempt.
A second fixture hides the same instruction in an attached document. The document can provide evidence about the export error, but it cannot assign roles or approve data transfer. Test whether source content is kept separate from trusted policy.
| Hostile variation | Required invariant |
|---|---|
| Claimed administrator role | No authority from untrusted text |
| Request for another customer’s data | Resource boundary enforced |
| Instruction in attachment | No execution-policy override |
| Request to send secrets externally | No permitted disclosure or send |
| Paraphrased malicious instruction | Control does not depend on one phrase |
Inspect tool attempts, returned fields, outgoing messages, and final records. Final-answer review alone misses actions performed before the assistant declines.
Keep legitimate support possible
A customer may quote hostile text as evidence of a problem. The workflow should not automatically reject the entire ticket because a suspicious phrase appears. Preserve the legitimate task while constraining effects through trusted controls.
Adversarial sets cannot prove immunity to every future attack. Use them as regression coverage for important boundaries and update from reviewed incidents. Limit tool scope so a successful model-level manipulation has fewer available consequences.
This test design is proposed. The voice integration case study describes related retrieval and tool work. Pair it with retrieved-document protections.
Hostile support input FAQ
Is filtering one dangerous phrase enough?
No. Attack wording can vary and ordinary quotes can contain the same phrase.
Does a safe final answer prove the test passed?
No. Verify disclosures and side effects throughout the execution.
What should the backend assume about model requests?
Validate them as untrusted proposals subject to current permissions, argument checks, and operation rules.