A knowledge-base update can change answers without changing the prompt or model. Evaluate retrieval, source selection, and business claims against the new knowledge version before allowing the update to handle live calls.
Knowledge changes affect more than factual wording
Knowledge-base evaluation checks whether the agent finds the right approved evidence and uses it accurately. Changes to documents, chunking, metadata, access rules, or indexing can alter what the model receives even when the visible source edit seems small.
In a proposed process, record the source snapshot and index configuration used by each test run. Build cases for changed facts, unchanged facts, removed guidance, and conflicting documents. Define which source is authoritative when several documents address the same question.
Keep permission tests in the suite. A document becoming searchable does not mean every caller may receive its contents. Retrieval scope should remain tied to trusted account and role context.
Test a fictional service-area revision
A business removes one postcode from its covered area. The candidate knowledge set should cause the assistant to stop offering service there while preserving answers for unchanged locations. An old document mentioning the postcode should not outweigh the current approved policy.
Add a caller who asks about the removed area indirectly, using a neighbourhood name. That fixture tests whether the retrieval pathway actually exposes the updated rule rather than passing only when the caller repeats the policy’s exact words.
| Knowledge change | Expected result |
|---|---|
| Covered area removed | No unsupported service promise |
| Opening hours unchanged | Existing answer remains accurate |
| Old policy still indexed | Current authority wins or ambiguity is surfaced |
| Required source unavailable | Agent states uncertainty or uses the approved fallback |
| Private document added | Unauthorised callers cannot retrieve it |
Inspect retrieved excerpts as well as final answers. A correct answer produced from memory can hide a broken retrieval pipeline and fail on the next related question.
Distinguish update defects from old coverage gaps
Run the baseline and candidate knowledge versions against the same fixtures. New failures show likely regression; failures in both may indicate missing coverage that predates the edit. Preserve both categories rather than attributing every issue to the latest update.
Full re-indexing can be expensive, and small updates may still affect neighbouring content. Use a targeted changed-fact suite plus a stable regression set. Expand the latter when a source change exposes an unexpected dependency.
This evaluation process is proposed. The voice platform case study describes knowledge retrieval work. Pair it with retrieved-instruction protections so updated sources cannot acquire execution authority.
Knowledge update FAQ
Is testing only the edited paragraph enough?
No. Also check retrieval, neighbouring questions, unchanged policies, and access boundaries.
What if two approved sources disagree?
Apply the defined authority rule or escalate uncertainty rather than choosing the more convenient answer.
Should old test expectations be overwritten?
Version them with the approved policy change so reviewers can distinguish an intentional change from a regression.