A developer says their AI system answers in under a second. Before treating that number as evidence for your product, ask what started the timer, what stopped it, and which requests disappeared from the calculation.
For founders reviewing engineering claims, a performance figure is useful only when its measurement conditions resemble the work you need done. A fast demonstration can be accurate and still tell you little about a busy production workflow.
What counts as production AI performance evidence?
Production evidence connects a defined workload to measured outcomes under a stated configuration. Request volume, model configuration, dependency behaviour, failure rate and observation window all affect the interpretation.
Ask the developer to separate measured results from estimates. An estimate can support an early planning decision, but it should include the assumptions that would change it.
For a voice system, distinguish time until speech begins from time until the requested appointment is actually saved. For retrieval, distinguish the speed of generating an answer from the quality and availability of its supporting sources.
Reconstruct one claimed result
Use this review worksheet with an anonymised example. The numbers belong in the supplied evidence rather than in a polished slide alone.
| Review question | Evidence to request | Why it matters |
|---|---|---|
| What was timed? | Named start and end events | Exposes omitted tool work |
| Which requests counted? | Eligible requests and exclusions | Reveals failed-request filtering |
| How variable was it? | Distribution and sample count | Averages conceal long waits |
| What configuration ran? | Model, release and environment | Makes comparisons interpretable |
| What went wrong? | Failure categories and recovery | Connects speed to reliability |
An illustration from a staging load test should be labelled accordingly. Do not ask an applicant to disclose a previous employer's confidential logs. A redacted methodology, recreated experiment or explanation of known limitations can show sound judgment without exposing customer data.
Ask whether the comparison changes the hiring decision
If the candidate compares two implementations, inspect whether the workload and success criteria stayed the same. Caching common answers may improve one workload while doing little for new customer questions. Removing a verification step may reduce latency while increasing incorrect writes.
I would rather review a modest result with a clear failure analysis than a large improvement with no denominator. The engineering skill is knowing which measure represents the customer experience and which optimisation has simply moved the cost elsewhere.
My voice platform contribution is project context, not a substitute for workload-specific performance evidence. Ask every candidate to make the same distinction.
Prepare a bounded verification exercise
Choose a representative workflow, agree on safe test inputs and define the evidence before the exercise starts. Keep the task small enough to inspect the resulting records and failures. Use the paid engineering trial checklist if code or instrumentation work is required.
Is a public benchmark enough to evaluate a developer?
It can establish context, but it does not demonstrate their implementation or your workload.
Must candidates share private production dashboards?
No. Request a lawful redacted example or a recreated experiment with an explicit limitation.
Which metric should we choose first?
Start with the completed customer action and define its timing and correctness criteria in your technical review brief.