A pilot without stop criteria can absorb months of engineering while every disappointing result is explained away. Decide in advance what evidence would justify continuing, narrowing, pausing, or ending the experiment.
The criteria should test the product hypothesis, not punish the team for learning. A failed assumption can be a useful outcome when the pilot exposes it before a larger investment.
Separate demand, usability, and operating criteria
Demand concerns whether the intended audience has the problem and chooses the workflow. Usability concerns whether participants can complete it. Operating criteria concern whether the team can support the promised outcome without unacceptable errors or hidden manual effort.
Write a review date, evidence source, owner, and decision rule for each. Avoid a single sign-up target that can be reached through founder persuasion while nobody returns to use the product.
Define a fictional research-study pilot
Imagine a product helps founders collect customer feedback through a shared link. The pilot hypothesis is that invited founders can create a short study, obtain relevant responses, and use the evidence to change or confirm a product decision.
Choose illustrative thresholds with the team, clearly labelled as planning assumptions rather than industry benchmarks. For example, require a specified number of independently completed studies, a documented review outcome from each participating founder, and an operating burden that fits the team's available support time.
Define stop conditions separately: participants cannot understand the questions despite a revision; founders repeatedly need direct staff help to publish; private responses become accessible to the wrong account; or the team cannot resolve a critical storage failure. Some conditions require immediate containment, while weak demand may require a scheduled review of audience and proposition.
Record explanations without moving the goalposts
Keep an experiment log of recruitment, product changes, outages, and manual interventions. If a provider outage disrupts the study, the team may reasonably repeat the affected period. Record that decision and its reason rather than silently extending the deadline until the numbers look acceptable.
Stop criteria also need nuance. A narrow audience may reveal strong value while a broader group sees none. That supports a scope revision, not necessarily either a full launch or total abandonment. State the revised hypothesis and run a new bounded experiment.
The tradeoff is discipline versus flexibility. Rigid thresholds can ignore useful qualitative evidence, while endless exceptions erase the experiment. Require a written decision that cites the evidence and names the next uncertainty.
StackFlight's portfolio account demonstrates a shipped study workflow. It does not claim a particular pilot conversion rate, revenue level, or validation outcome.
Questions about pilot stop criteria
Should the founder choose targets alone?
The founder owns the product decision, but engineering and support should review whether the evidence can be collected and the operating limits are realistic.
Does missing a target mean the product must close?
Not automatically. It requires the agreed review. Narrowing the audience or changing the workflow can be appropriate when supported by the evidence.
What should the review produce?
Produce a continue, revise, pause, or stop decision with reasons, owners, and a next date. Bring unresolved technical assumptions to a pilot review.