Department-based search tests should verify permitted retrieval and forbidden disclosure. Use fictional employees with overlapping roles so the evaluation catches both missing access and accidental cross-team exposure.
Permission testing needs positive and negative cases
A departmental permission matrix maps trusted employee relationships to documents and actions. It should cover resource reads, snippets, previews, citations, caches, and answer generation rather than only the original file store.
For a proposed test set, create fictional sales, engineering, support, contractor, and administrator identities with explicit expected scope. Use synthetic restricted markers in documents. Keep the model and retrieval configuration fixed while varying the trusted identity.
Do not infer department from the query text. A user asking “as an engineer” should not gain engineering access. The application’s identity context supplies the relevant relationship.
Build a fictional shared-policy matrix
All employees may read a general leave guide. Engineering can read a technical incident library. A limited support group can read customer diagnostic notes. A manager has a scoped exception, not blanket access to everything.
Ask identical and paraphrased questions under every identity. The expected answer may differ because eligible evidence differs, but the assistant should not reveal that a restricted document exists merely through a suggestion or citation title.
| Test identity | General guide | Incident library | Scoped customer notes |
|---|---|---|---|
| Sales employee | Allow | Deny | Deny |
| Engineering employee | Allow | Allow | Deny |
| Assigned support employee | Allow | As explicitly granted | Scoped allow |
| Contractor | Explicit subset | Explicit subset | Deny by default under this fixture |
This table is a fictional fixture, not a universal company policy. The actual organisation must approve its relationships and expected decisions.
Test transitions as well as static roles
Remove a group, change teams, expire a contract, and open an earlier citation. Verify new requests and derived artifacts under the defined propagation policy. A static matrix can pass while revocation remains broken.
Measure false denials and leakage separately. A system that denies everything can appear secure while failing its useful purpose. Positive cases prove employees can access the intended sources, and negative cases prove the boundary is enforced.
This evaluation is proposed. The voice integration case study describes related retrieval work without claiming these tests already exist. Pair it with release permission tests.
Department search FAQ
Should every department have one flat role?
Only if that matches actual policy. Overlapping projects and resource relationships may require more detail.
Is final-answer review enough?
Inspect candidates, assembled context, metadata, previews, and tools as well.
What proves usefulness and protection together?
A reviewed matrix of allowed and denied cases, tested across normal requests and access changes, provides complementary evidence.