A benchmark built from tidy policy pages will tell you how the assistant handles tidy policy pages. It may reveal little about the scanned contracts, repeated filenames, and spreadsheet footnotes employees actually search.
Choose representative documents by the decisions they support and the failures they can cause. A useful benchmark is a deliberately sampled evaluation set, not simply a folder of files that were convenient to export.
Sample the document population along meaningful boundaries
Start with the employee workflows: finding an entitlement, resolving an exception, comparing versions, or locating a number. For each workflow, inventory formats, document sizes, ownership, access groups, update frequency, and terminology. Keep those labels alongside the evaluation cases.
Include common documents in proportion to their operational importance, then deliberately add rare but costly edge cases. Report those groups separately. An oversampled exception set is useful for stress testing but should not be described as the expected production traffic mix.
Build a fictional service-team benchmark
Imagine a support team uses product manuals, customer entitlement sheets, internal escalation notes, and archived release notices. Select examples from each category. Add one manual with a table, one superseded notice, one restricted escalation note, and two documents with the same title but different products.
Write questions after inspecting the chosen material. For each, preserve the expected source spans, applicable revision, reader permissions, and the minimum answer. Include an entitlement question that cannot be answered because the customer's current plan is absent.
Now inspect the proposed set for shortcuts. If every question repeats the document heading verbatim, add employee-style paraphrases. If every correct source is a single paragraph, add a question requiring a heading and a footnote. If all restricted documents belong to one department, add a second permission pattern.
These examples are a proposed sampling exercise. The counts you choose should follow your corpus and review budget, not an arbitrary claim that a particular sample size proves reliability.
Preserve a held-out set and document its limits
Separate cases used to tune chunking or prompts from cases reserved for comparison. Otherwise, repeated optimisation can turn the benchmark into a set the system happens to fit. Keep document revisions fixed within a comparison and record when the benchmark itself changes.
Human labels also have limits. A source can be incomplete or internally inconsistent. Record ambiguous questions rather than forcing a confident reference answer. A good evaluation set should reveal documentation defects as well as model defects.
The official retrieval evaluator guidance describes evaluation against relevance labels. Your labels still need domain review. See company document integration and the Nucleus portfolio account for related context.
Questions about benchmark sampling
Should the benchmark contain every document?
No. Sample deliberately, track coverage, and add cases as new failure categories appear. Document-level completeness is not the same as workflow coverage.
Are synthetic documents useful?
Yes, for controlled exceptions and permission tests. Keep them labelled so their results are not mistaken for evidence about the full production corpus.
When should the benchmark change?
When the product scope, corpus, or risk changes. Preserve prior versions so a technical review can distinguish a stronger assistant from an easier test set.