Small on-device models are easier to evaluate when the task has clear inputs, useful outputs, and a way to recognise unsupported results. Start with a narrow assistance workflow rather than promising an unrestricted expert.
Task fit is empirical. A short prompt does not automatically make a task easy, and a structured response can still contain incorrect facts.
Define the job and failure boundary
Choose one user goal, source type, output structure, and human-review step. Identify consequences of a wrong result and whether the action is reversible. Keep consequential decisions separate from generated suggestions unless the product has stronger evidence and controls.
Review current model documentation and runtime support for the exact candidate. Use published capabilities to plan tests, not to skip them.
Evaluate a fictional paragraph-to-checklist task
Suppose a business app turns an authorised procedure paragraph into a draft checklist. The expected output preserves each required step and condition, adds no unsupported obligation, and flags missing information.
Test a clean procedure, an exception, a negative instruction, a table-dependent step, and incomplete text. Review every proposed checklist item against the source. A plausible extra step should not earn credit merely because it sounds helpful.
Compare with a simple non-generative baseline such as displaying the original procedure with manual checkboxes. The AI feature needs to improve a meaningful part of the workflow, not just produce a different format.
Use constraints that preserve usefulness
Bound source range and output length, provide required vocabulary where justified, and define an unsupported-answer path. Avoid arbitrary truncation that removes the condition the task depends on.
Measure correction effort, completion, waiting, memory, and failure recovery on audience-representative hardware. No benchmark or claim that a particular small model passes this task is made here.
The tradeoff is narrow reliability versus breadth. A constrained feature can be useful and easier to explain, while an open-ended assistant may appear impressive but create a much larger evaluation and support burden. Expand only when evidence supports the next task.
The ChalkLens portfolio account documents a bounded local teaching workflow. This procedure-checklist example is fictional and does not describe its current feature set or measured model performance.
Retain rejected examples so later improvements face the same difficult cases.
Questions about small-model task selection
Are structured outputs automatically correct?
No. Structure helps parsing and review, while source support and semantic correctness still need evaluation.
Should the smallest model be chosen first?
Use candidates that fit the requirement and hardware constraints, then compare actual task outcomes. Size alone is not a selection rule.
What should the first pilot prove?
Prove one useful complete task, its unsupported cases, correction effort, and resource fit. Bring the evidence to a local AI task-scoping review.