A custom model cannot establish whether customers need the workflow. Before investing in model development, test whether the proposed result solves a recurring problem and whether users will accept the surrounding review process.
Separate product uncertainty from model uncertainty. Customers may reject a technically impressive output because it arrives too late, lacks a required source, or creates more checking work than it saves.
Define the task and a non-AI baseline
Choose one user, one input type, and one useful output. Record how the user completes the task today, including tools, handoffs, waiting time, and mistakes. Ask about a recent instance rather than a hypothetical willingness to use AI.
Build a baseline using the simplest authorised method that can test the workflow. That may be manual review, rules, search, or an existing model. Label manual assistance clearly and obtain appropriate consent before sending real data to any external service.
Test a fictional supplier-document assistant
Imagine a founder proposes a custom model to extract renewal dates from supplier agreements. Start with a small, authorised set of representative documents and a reviewer-defined output: the date, supporting passage, relevant condition, and an uncertainty flag when the date cannot be established.
Have a person establish the reference answer. Compare the current manual process with the baseline assistant. Record whether the reviewer accepts, corrects, or rejects each proposed result, and why. A wrong date with a convincing explanation is more serious than a clear request for review.
Include a document with no renewal clause, one with several dates, and one whose term depends on another agreement. These cases distinguish useful extraction from plausible guessing. The exercise is illustrative; it provides no measured accuracy or time-saving claim.
Decide whether the bottleneck is actually the model
Inspect failed cases. If the relevant page never reaches the model, improve ingestion or retrieval. If the source is ambiguous, a custom model cannot create the missing business fact. If reviewers spend time on an awkward interface, changing the model may leave the real cost untouched.
Custom development becomes a better question when you have a stable task, a reviewed dataset you are permitted to use, a meaningful baseline gap, and a way to evaluate improvements. It also creates training, deployment, monitoring, and maintenance work that belongs in the decision.
The tradeoff is control versus operating burden. A managed model can help test demand quickly but may not fit data, cost, or latency constraints. A custom model may address a specific gap but cannot be justified by assuming all failures are model failures.
The StackFlight product account describes collecting customer evidence before product decisions. Its current workflow does not automatically process feedback with AI, and this extraction example is separate.
Questions about custom-model validation
Is strong customer interest enough to start training?
It is useful evidence, but you also need a defined task, data rights, a baseline, and measurable failure categories.
Should the first test use private customer documents?
Only with appropriate authorisation and handling controls. Controlled fixtures can establish the workflow before sensitive data enters the experiment.
What should the decision packet contain?
Include the user problem, baseline outputs, correction reasons, data constraints, and remaining uncertainty. Use it for a technical feasibility discussion.