An employee asks about the current leave policy, and the knowledge assistant retrieves a clear answer from last year's handbook. The retrieval was relevant to the question but wrong for the time and organisation.
Document relevance needs version and applicability rules, not just semantic similarity.
Store policy identity separately from file identity
A policy can have several files, revisions, drafts, and regional variants. Give the logical policy an identity and record which approved version applies to which audience and time period.
For a proposed internal system, useful metadata includes the policy owner, approval status, effective date, superseded version, relevant organisation or region, and source reference. Do not use upload time as a substitute for effective date.
Filter eligibility before ranking passages
For a fictional leave-policy question, establish which policy versions the user is allowed to read and which ones are applicable to the question.
| Question context | Retrieval behaviour |
|---|---|
| Current policy | Prefer the applicable approved version |
| Historical date | Use the version effective at that time |
| Region not established | Ask or restrict the answer's scope |
| Conflicting approved records | Expose the conflict for review |
Keep access and applicability separate. A user can be permitted to read an old policy without that policy being the correct answer to a current question.
The company-document integration guide covers the broader source boundary. Version selection is a specific retrieval rule within that architecture.
Keep the selected version attached to the answer
Return the document version and effective context with each retrieved passage. The answer should cite the actual source it used, not a generic link that silently opens a newer revision.
If the source system always serves the current file at one URL, consider how a reviewer can inspect the version that supported an earlier answer. Retention and access choices need to match the organisation's requirements.
Do not ask the model to resolve contradictory policy documents as though it had authority to set company policy. A document owner must decide which record governs.
Test the same question across policy changes
Use fictional versions with a deliberate difference in one rule. Ask the current question, a historical question, a region-specific question, and a question during a staged policy rollout.
Inspect retrieved passages before judging the generated answer. If the old version entered context, changing answer wording may hide the symptom without fixing selection.
Explicit metadata creates maintenance work. For a small pilot, one curated policy collection with a named owner may be more practical than connecting every shared drive immediately. The knowledge-sync article covers keeping the indexed copies current.
Policy-version questions
Is the newest uploaded document always authoritative?
No. Upload time, approval, and effective date mean different things.
Should old versions be deleted automatically?
Historical questions may need them. Apply an explicit access and retention policy.
What should the first test show?
Two versions, two effective dates, and a traceable source choice. Include that requirement in a knowledge-system brief.