Incomplete textbook text should produce a visible scope limitation, not an apparently complete lesson built from guesses. Test missing evidence deliberately so the product can distinguish extraction failure from a genuinely short source.
Use controlled authorised fixtures and preserve the original page. This lets reviewers identify exactly what the model received and which claims it could legitimately support.
Define failure cases by lost meaning
Remove a condition, caption, definition, table header, answer paragraph, or section transition from a known source copy. Keep the test change labelled so it cannot be mistaken for the real textbook.
Write expected behaviour for each gap. The app may ask for a better source, narrow the lesson, show a warning, or generate only the supported parts. The correct choice depends on the task and the importance of the missing material.
Test a fictional chapter with an omitted exception
Suppose the original passage states a general rule and one exception. The extracted fixture omits the exception while retaining the surrounding paragraph. A draft that turns the general rule into an absolute statement should fail the review.
Compare the incomplete and complete versions under the same task. Review source support, omitted facts, invented detail, warning clarity, and whether the teacher can find the affected page. A correct answer based on general model knowledge should not automatically pass a source-grounded test.
Add a missing quiz-answer passage and a scrambled table. Verify the system does not manufacture an answer key or numeric comparison from absent structure.
Keep diagnosis linked to the extraction stage
Record source identity, extracted revision, selected range, generated draft, and review notes together. A model prompt change cannot reliably repair text that never reached the model.
Make the teacher-facing limitation actionable. “Some content may be missing” is weaker than naming the affected page or section and offering correction or a narrower selection. Avoid overwhelming users with uncertain warnings on every clean page.
The tradeoff is caution versus usefulness. The product should still support a valid narrow task when enough evidence exists, while refusing or qualifying unsupported conclusions. Evaluate the balance using representative teacher tasks rather than scoring refusal frequency alone.
The ChalkLens portfolio account documents source-based local teaching functionality. This omitted-exception matrix is proposed guidance and does not claim a tested evaluator, accuracy result, or classroom study.
Questions about incomplete source tests
Can a fluent answer still fail?
Yes. Fluency does not establish that the supplied source supports the answer or that important conditions were preserved.
Should all generation stop after any extraction gap?
Not necessarily. Identify whether the gap affects the requested objective and define a supported reduced scope where appropriate.
What should a bug report include?
Include original page, controlled extracted fixture, missing span, task, actual draft, and expected limitation. Bring the packet to an education AI evaluation review.