A developer can produce a convincing answer about testing while showing only a happy-path demo. To assess their debugging approach, give them a concrete failure and ask what evidence would distinguish several possible causes.
The useful skill is narrowing uncertainty without making uncontrolled changes to a live system.
Use one failure with several plausible layers
For a fictional AI booking flow, the caller corrects the date, the agent acknowledges it, and the old date reaches the calendar.
| Layer | Evidence the candidate might inspect |
|---|---|
| Conversation | Was the correction captured accurately? |
| Application state | Which request version remained active? |
| Tool arguments | Which values crossed the integration boundary? |
| Provider result | What operation actually succeeded? |
| Final record | What will staff and customers see? |
Look for a reproducible hypothesis and a proposed isolation step, not a reflex to replace the model or add a longer prompt.
Ask how the test could fail despite a fluent answer
A useful test plan defines expected business state independently of the AI's explanation. It can include an unchanged record when the action is denied, or a visible pending state when the result is uncertain.
Ask how fixtures avoid real customer messages and how the engineer cleans up only the records the test created. Testing discipline includes the environment boundary.
The saved-record assertion article provides an example of that separation.
Evaluate proposed changes and trade-offs
The candidate should explain what they would change, what remains uncertain, and how they would verify that the fix did not break adjacent behaviour.
For slow workflows, ask for a measurement boundary and workload. A claim that the system is faster needs context; a single stopwatch observation may not describe the calls customers actually make.
For intermittent faults, look for identifiers and traces that connect the interaction to the final operation. Collecting every log line without a correlation strategy can make the investigation slower.
Assess communication as part of debugging
Ask for a short report for a nontechnical owner: observed problem, confirmed cause or current hypothesis, affected scope, next action, and acceptance evidence.
An honest statement that access is missing is more useful than presenting a plausible theory as a verified diagnosis. A strong engineer can request the minimum evidence needed without demanding unrestricted production access.
Use a bounded paid exercise when you need to see the process in practice. Avoid a broad unpaid production repair disguised as an interview.
Debugging-assessment questions
Is a large test count enough evidence?
Inspect what the tests assert and which important failures remain uncovered.
Should every bug require a complete rewrite?
Ask for evidence connecting the architecture to the observed failure.
What should the next step be?
Agree on one reproducible case and its expected outcome for a technical review.