Permission tests should be a release gate for a knowledge assistant, not a one-time project checklist. Search, caching, ingestion, identity, and source-preview changes can each alter what protected information reaches a user.
A permission suite protects resource invariants
Release permission testing verifies allowed and denied behaviour across trusted identities and resource scopes. Include both useful positive access and negative disclosure cases; denying everything is not a successful service.
For a proposed suite, maintain fictional employees, groups, documents, client namespaces, and synthetic restricted markers. Pin source and configuration versions. Test retrieval context, suggestions, previews, citations, exports, and caches under the relevant policy.
Do not restrict the assertion to final answer text. A marker entering an unauthorised model request is a boundary failure even if the final response happens not to repeat it.
Build a fictional release gate
A candidate changes vector-index filtering. The suite asks the same policy question as an ordinary employee, a project member, a contractor, and an administrator with limited content access. Each receives only its reviewed permitted scope.
The candidate also runs revocation-during-query and re-index activation fixtures. A static role matrix can pass while a cache or rebuild restores stale access. Those transition failures should remain visible as critical events under the predefined release policy.
| Release change | Relevant fixture |
|---|---|
| Retrieval query rewrite | Trusted scope survives generated filters |
| Cache optimisation | No cross-user or revoked-source reuse |
| New connector | Source permission and namespace preserved |
| Preview interface | Titles and snippets follow access |
| Index rebuild | Incomplete metadata cannot broaden scope |
Run the suite against the deployable configuration, not an earlier local approximation. Preserve safe failed artifacts for diagnosis and compare baseline failures separately from newly introduced ones.
Keep gates meaningful and maintainable
A noisy gate can encourage bypasses. Tie each critical assertion to a concrete disclosure or denial consequence and keep fixtures reproducible. Update expected policy only through an authorised reviewed change.
Automated checks cannot cover every future interaction. Use monitored rollout, reviewed samples, and incident feedback to expand the suite. A passing release establishes the tested boundaries under the tested conditions, not immunity to all access failures.
This release process is proposed. The voice integration case study describes related integration work without claiming this test infrastructure. Pair it with department permission matrices.
Permission release FAQ
Should a relevance gain override a leakage failure?
Follow the predefined blocking policy; do not average away a material access violation.
Are positive cases necessary?
Yes. Verify that authorised employees can still perform the supported work.
What should trigger suite expansion?
Material architecture changes, incidents, new source systems, and demonstrated gaps in the existing coverage.