A company assistant performs well on generic benchmark questions and still fails the questions employees ask every week. The evaluation set did not represent the organisation's documents, vocabulary or access boundaries.
For teams building internal knowledge tools, construct a reviewed set from relevant company questions and controlled source evidence. The set should help identify failure causes rather than produce one reassuring score.
What belongs in a company-question evaluation set?
Include the question, appropriate user role, relevant source version, expected supporting evidence and acceptable answer boundaries. Some questions should have no supported answer or require clarification.
Use appropriate permission and privacy controls when adapting real questions. Remove unnecessary personal or confidential detail while preserving the task structure the evaluation needs.
Design purposeful coverage
The proposed worksheet below shows categories for a fictional internal-policy assistant. It is a coverage aid, not a universal dataset size recommendation.
| Question category | Expected behaviour |
|---|---|
| Common documented policy | Correct scoped answer with support |
| Department-specific rule | Appropriate source and user access |
| Ambiguous terminology | Clarification or explicit interpretation |
| Missing supporting source | Honest evidence gap |
| Outdated policy version | Current-source precedence under defined rules |
| Permission-restricted answer | No unauthorised retrieval or disclosure |
Avoid filling the set with minor paraphrases of one easy question while leaving those behavioural boundaries untested.
Separate development examples from review evidence
If the team repeatedly tunes against the same fixtures, the set becomes development material. Keep an appropriately managed held-out review subset or add newly reviewed cases to check whether changes generalise beyond the examples used for tuning.
Record expectation changes and their reasons. A policy update can legitimately change an answer, but quietly relabelling a failed case makes the performance history difficult to interpret.
Inspect the pipeline at useful boundaries
Review extracted source quality, retrieval, permission enforcement, answer support and final user wording separately. A correct answer guessed without the expected evidence is not proof that retrieval worked.
The citation-support guide helps define the source relationship. The parser fixture guide covers upstream extraction failures that can affect the same question.
My knowledge platform contribution supplies related context, not a claim that this exact company dataset exists. Each organisation needs its own reviewed questions and source boundaries.
Keep the evaluation manageable and purposeful. Add cases from observed incidents and material workflow changes rather than chase an arbitrary total that increases maintenance without improving coverage.
Can generic public benchmarks replace company questions?
They can provide context, but they do not establish behaviour on your documents, roles or vocabulary.
Should every question have a confident answer?
Include unsupported and ambiguous cases so appropriate limits are evaluated too.
What should we prepare first?
Collect a small reviewed set of real task patterns with safe sources for the knowledge evaluation discussion.