A spreadsheet comment saying “wrong policy” loses value when nobody can recover the answer or source revision it refers to. Human evaluation notes need stable links to the exact test execution, not just the current question text.
For teams iterating on retrieval and prompts, this is a provenance problem. A note should explain what the reviewer saw, why it failed, and which later change addressed that failure.
Separate the test definition from the execution
Give each test case a stable identifier. Its definition includes the question, reader identity, expected behaviour, source references, and rubric version. Give each run a different identifier that records the assistant version, retrieval configuration, source snapshot, time, answer, and retrieved passages.
Attach review notes to the run. If you edit a question or change its expected answer, create a new case revision. Do not rewrite historical notes to match the new expectation. That would make previous release decisions impossible to reproduce.
Trace a fictional approval-policy defect
Imagine case PUR-17 asks whether a fictional purchase needs finance approval. Run A omits a source-stated threshold exception. A reviewer marks the exception span and records “incomplete decision condition.” Another reviewer initially passes the answer because the threshold is correct.
Keep both judgements and their reasons. A short calibration discussion establishes that the exception is mandatory. Record the adjudication against the rubric version rather than deleting the disagreement. The disagreement showed exactly where the review standard was unclear.
After a retrieval change, run the same case again as Run B. Link the fix to the prior failure, then check whether the exception is now retrieved and correctly used. A new note saying “looks better” is insufficient. Identify the expected condition and whether the answer includes it.
Make notes usable for engineering work
A compact review record needs the affected claim, source span, failure category, severity, reviewer, decision date, and next owner. Add an issue link when repair work is scheduled. Keep raw private source material behind its own access controls rather than copying it into broadly shared tickets.
Stable links create operational overhead: records require retention, permissions, and a migration plan when evaluation tools change. Start with a simple table and durable run artefacts if your team is small. A complex annotation system is unnecessary if reviewers cannot consistently use a basic rubric.
Review notes are evidence of judgement, not unquestionable ground truth. Preserve uncertainty when the source itself is unclear. Revisit those cases when the document owner changes the policy.
The citation guide explains source support, and the Nucleus portfolio page gives related context for knowledge integrations.
Questions about evaluation records
Can we attach notes to the question alone?
Use the question for general guidance, but attach answer-specific judgements to a run. Different versions can produce materially different failures for the same question.
Should reviewers agree before scoring the whole set?
Calibrate on a shared sample first. Keep disagreements visible and resolve changes to the rubric explicitly.
What is the first migration step?
Export case IDs, run IDs, notes, source references, and rubric versions together. Check that one old failure remains reproducible before planning a wider review workflow.