A fluent answer can confidently omit the only exception that matters. Evaluating a knowledge assistant therefore starts with what the employee needed to decide, then checks whether the answer supplied authorised, supported information for that decision.
For product owners, a useful evaluation distinguishes retrieval quality, answer evidence, and practical task completion. Combining them into one impression score makes failures hard to repair because a missing source and an invented conclusion demand different changes.
Define an answer contract before reading responses
Write each test as a question, an authorised reader identity, the source revisions available, the minimum facts required, and the actions the assistant must avoid. Include questions that have no supported answer. Otherwise, an assistant that guesses on every request can appear more helpful than one that recognises a genuine evidence gap.
The official RAG evaluator documentation distinguishes retrieval and response evaluation. Use that distinction to organise your own rubric rather than treating a vendor score as your product's acceptance decision.
Score a fictional purchasing question
Imagine an employee asks whether a 1,200-unit purchase needs approval. The fictional policy says purchases above 1,000 require a manager, except approved catalogue items. A second document lists the requested item in the catalogue.
An answer saying “yes, manager approval is required” is readable and cites a real threshold. It still fails because it missed the exception. An answer saying “no approval is ever required for catalogue items” adds an unsupported universal rule if other approval conditions remain in the policy.
Make a review row for the threshold, the exception, the item's catalogue status, and the conclusion. Mark each required fact as present, absent, or contradicted. Separately record unsupported additions. Ask the reviewer whether the employee can proceed or must still resolve a material ambiguity.
Use a small operational test matrix
Test the original question, a paraphrase without the policy's exact terms, an outdated source, conflicting documents, a restricted-reader account, and a missing-item case. Preserve the retrieved passages for every response so reviewers can tell whether retrieval or synthesis failed.
Run more than once when outputs vary. Report the number of cases and observed failures instead of presenting a small pass percentage as a guarantee. A test set containing only clean policy pages says little about spreadsheet footnotes or inconsistent department terminology.
Human review costs time, and reviewers can disagree. Calibrate on a few shared examples and retain disagreements as evidence of unclear policy or unclear scoring. Automated judging can prioritise review, but its rationale also needs spot checks.
See the knowledge integration overview for surrounding architecture and the Nucleus project for portfolio context.
Questions about assistant quality
Should writing quality count at all?
Yes, when ambiguity or poor structure blocks use. Score readability separately so polished wording cannot compensate for an incorrect decision.
What is the most valuable first benchmark?
A small set of real employee tasks with reviewed evidence and explicit failure conditions. Include exceptions and questions that should remain unanswered.
When is the evaluation ready for a release decision?
When someone owns each material failure and the team has agreed which failures block release. Use those cases to frame a technical review, with private source data removed.