01 /Tag — Evaluation
19 articles in this topic.
Build knowledge-assistant release gates around permission boundaries, critical task failures, reproducible source evidence, and a clear rollback decision.
Choose RAG benchmark documents by workflow, format, age, permissions, and failure risk so clean policy pages do not hide retrieval weaknesses.
Compare knowledge-assistant versions with paired cases, fixed source snapshots, repeated runs, and separate reporting for critical regressions.
Compare raw and cleaned dictation by meaning preservation, critical facts, structure, review effort, and final task usefulness with a fixed authorised sample and transparent settings.
Build a citation-version audit that checks document identity, effective dates, source spans, and the actual destination a reader opens.
Evaluate knowledge assistants with claim-level evidence, task completion, permission tests, and explicit abstention cases rather than polished language.
Evaluate table-based retrieval by preserving row labels, column headers, units, footnotes, and revisions before judging the final numeric answer.
Keep human evaluation decisions reproducible with stable test identifiers, immutable run records, source revisions, and explicit reviewer disagreement.
Measure completeness with required evidence-backed facts, explicit optional context, and penalties for unsupported additions using a worked policy example.
Interpret citation reopens using task context and reason codes so healthy verification is not confused with incomplete or untrustworthy AI answers.
Measure dictation productivity through equivalent tasks, complete timing, critical-error review, setup and repair effort, and repeated samples rather than speaking speed alone.
Review multi-document AI answers by separating supported facts from inferred joins, mismatched scopes, and conclusions that no cited source establishes.
Separate local-model potential from application guarantees through task-specific evidence, bounded feature promises, supported inputs, human review, and clear measurement limits.
Test permission changes across retrieval, cached answers, conversation history, and citation destinations with a repeatable before-and-after access exercise.
Test conflicting knowledge sources with explicit authority rules, effective dates, and unresolved cases instead of allowing the assistant to pick a convenient passage.
Test teaching AI with deliberately incomplete extraction to verify gap recognition, supported output scope, teacher warnings, and preservation of original source evidence.
Test curriculum vocabulary with source-backed definitions, near-neighbour terms, age-appropriate paraphrases, OCR variants, and reviewed lesson and quiz outputs.
Evaluate entry-level on-device AI using named hardware, realistic documents, cold and repeated runs, task quality, resource pressure, and honest limits instead of flagship extrapolation.
Turn unanswered knowledge questions into owned documentation work by distinguishing source gaps from retrieval, permission, and ambiguity failures.