An answer can be incomplete in fifty words or misleading in five hundred. Rewarding length gives a knowledge assistant an incentive to add plausible details even when the source does not support them.
Define completeness as coverage of the facts needed for the user's task. Keep support for those facts as a separate requirement. A response should not earn extra credit for statements the reader never needed and the documents cannot establish.
Write a fact inventory before scoring answers
For each evaluation question, list required facts, required conditions, acceptable uncertainty, and optional explanations. Give each required item a source span and revision. If reviewers cannot agree on the inventory, resolve that disagreement before treating an automated score as meaningful.
Use the distinction between groundedness and completeness in the official RAG evaluator reference as a starting vocabulary. The scoring scheme below is a proposed operational rubric, not a claim about any evaluator's default behaviour.
Score a fictional return-policy answer
Suppose a fictional policy requires a return within 21 days, a receipt, and unopened packaging. The customer asks how to return a sealed item purchased 12 days ago. The required inventory includes eligibility subject to proof of purchase, where to start the request, and any source-stated next step. It does not include a refund timetable unless the policy provides one.
Answer A explains the return channel and receipt requirement but omits unopened packaging. Answer B covers all required conditions and adds “refunds always arrive within two days.” Answer C covers the required conditions and says the source does not specify refund timing.
Mark A incomplete. Mark B as covering the inventory but failing evidence support because of the invented promise. Mark C complete for this task, provided its channel and conditions match the sources. Avoid averaging B's unsupported promise away with several correct sentences.
An illustrative scorecard could report four required facts covered out of five alongside one unsupported claim. Those counts describe the example; they are not measured product results or a universal release threshold.
Check that the rubric does not punish useful brevity
Test short and long paraphrases of a correct answer. Both should pass when they preserve the conditions and next action. Add a deliberately verbose answer with irrelevant policy history. It should gain no completeness credit for that history.
Some questions require context that the user has not supplied. Count a targeted clarification as appropriate completion of the current turn when the answer contract requires it. Otherwise, the benchmark encourages the assistant to guess missing facts.
Related discussion appears in the source citation article, with portfolio context in Nucleus.
Questions about completeness scoring
Should every fact have equal weight?
Not necessarily. A missed safety condition or approval requirement may be more serious than omitted background. Choose weights before reviewing candidate answers.
Can optional detail reduce quality?
Yes, when it introduces unsupported claims, hides the decision, or makes conditions harder to find. Optional does not mean exempt from evidence review.
How should a team start?
Write fact inventories for five representative questions and compare two reviewers' scores. Use disagreements to improve the rubric before requesting a wider evaluation review.