When two documents disagree, the assistant needs a policy for resolving the disagreement. Returning whichever passage ranked first turns a retrieval accident into an apparent business rule.
Product owners should define when the system can resolve a conflict and when it must surface uncertainty. The model cannot reliably infer your organisation's document authority from confident wording or a familiar filename.
Describe authority and applicability separately
Authority asks which source owns the rule. Applicability asks whether that rule covers this reader, product, region, and date. A newer document may be authoritative for one region while an older document remains applicable elsewhere.
Attach reviewed metadata where possible: owner, status, effective date, scope, and superseded revision. Treat missing metadata as missing evidence. A statement that looks official is not a replacement for an agreed source hierarchy.
Test a fictional remote-work exception
Suppose a company handbook says remote work requires manager approval. A department page says employees can work remotely on Fridays. A later operations notice suspends Friday remote work for one office during an invented maintenance period.
Ask about a Friday inside that period, a Friday outside it, and an employee from another office. The reference answer must identify which scope and dates apply. If the sources do not establish whether the department exception overrides the handbook, the expected response should state the conflict and point to its owner.
Build paired cases that change only one fact. Reverse the order of retrieved passages. Remove the operations notice. Replace its effective date with an upload date. The answer should follow the evidence and authority policy, not the order in which the model sees text.
Record the conflicting spans even when the final answer is correct. Otherwise, a reviewer cannot tell whether the assistant resolved the disagreement or happened to choose the preferred sentence.
Score recognition before resolution
Use separate fields for conflict detected, applicable scope identified, resolution supported, and escalation actionable. An answer that correctly recognises uncertainty may be more useful than a clean but unsupported yes or no.
There is a usability cost to excessive caution. If every minor wording difference triggers escalation, staff will stop trusting the assistant for routine tasks. Calibrate on disagreements that materially change an action, entitlement, obligation, or deadline. Cosmetic variations can be handled without turning every answer into a policy dispute.
Conflicts can also signal documentation debt rather than a model defect. Send unresolved cases to the source owner and retain them in the benchmark until the authoritative rule is clarified. The company document guide covers related integration boundaries; the Nucleus case study describes relevant portfolio work.
Questions about conflicting sources
Should the newest document always win?
No. A newer upload may cover a different scope or contain an unapproved draft. Verify authority, status, and effective date together.
Can the assistant quote both sources?
Yes, if it clearly explains the disagreement and avoids implying that quoting both resolves it. Provide a useful next owner or clarification request.
How do we know a repair worked?
Repeat paired cases with passage order changed and scope held constant. Bring unresolved failures to a knowledge integration discussion with source identities preserved.