An employee asks about a policy that has never been documented, and the assistant supplies a plausible answer with an unrelated citation. The missing source has become an invisible assumption.
For teams evaluating knowledge assistants, include questions that the available evidence cannot answer. A system's ability to acknowledge that limit matters alongside its performance on documented facts.
What is an unsupported-answer test?
It is a controlled case where the authorised source set lacks sufficient evidence for the requested claim. The expected behaviour may be clarification, an explicit evidence gap or an approved route to a responsible person.
Keep this distinct from a retrieval miss. If the relevant document exists and should have been accessible, failure to retrieve it is a different problem from a genuinely undocumented question.
Build missing-evidence cases deliberately
Use fictional company policies with known coverage. The proposed matrix separates several superficially similar situations.
| Evidence condition | Expected review boundary |
|---|---|
| No source contains the fact | Do not invent an organisational rule |
| Source is topically related only | Explain the unsupported detail |
| Required source is access-restricted | Do not disclose through a workaround |
| Question is ambiguous | Ask a relevant clarification |
| Retrieval misses an available source | Record a pipeline failure, not successful refusal |
Avoid rewarding every refusal as safe success. An assistant that declines all questions would pass a poorly designed unsupported set while failing its useful purpose.
Inspect why the answer was withheld or produced
Review the retrieved passages and source-access state. A correct-looking uncertainty response can still conceal a permission bug or missing ingestion job that should be repaired.
The company evaluation-set guide helps balance supported and unsupported cases. The citation-support review distinguishes a relevant-looking link from actual evidence.
Make the next step useful without promising unknown facts
If the business has an approved policy owner, provide that route appropriately. Do not name an arbitrary employee or claim a response time that the operating process has not established.
My knowledge platform contribution supplies related project context, not a claim that every unsupported question is handled correctly. New deployments need their own evidence-gap fixtures and owner review.
Record undocumented questions as potential documentation work when the organisation approves that process. Keep them separate from automatically generated policy: a question can reveal a need for a source without authorising the AI to create the rule itself.
Is a refusal always a correct result?
No. Determine whether evidence was genuinely absent, inaccessible by policy or missed by the pipeline.
Can general model knowledge fill an undocumented company rule?
It should not be presented as an authoritative organisational policy without the required approved source.
What should we test first?
Ask a plausible question deliberately absent from a fictional policy set in the knowledge assistant review.