A polished chatbot demo shows that someone can connect a model to an interface. It does not establish how they handle permissions, failed actions, conflicting evidence, or the operational work around the conversation.
Evaluate an AI engineer's portfolio by tracing one complete user outcome and the boundaries the engineer actually owned.
Ask what the project proves
Separate a prototype, independent product, client contribution, and operating production service. All can demonstrate useful work, but they support different claims.
| Portfolio evidence | Question to ask |
|---|---|
| Live interface | What can a user actually finish? |
| Technical explanation | Where are validation and permissions enforced? |
| Test material | Which failures were exercised? |
| Case study | What did this engineer personally contribute? |
| Outcome claim | What evidence supports the stated result? |
An engineer should be able to explain uncertainty and limitations without treating them as embarrassing omissions.
Trace one action beyond the generated answer
For a fictional booking product, ask how a preferred time becomes a confirmed record. Where is capacity checked? What happens if the write times out? Which office view shows unresolved work?
A good explanation can include trade-offs and a smaller first-release scope. A diagram alone is insufficient if the engineer cannot connect it to actual behaviour or clearly label it as a proposal.
Inspect ownership and maintainability
Ask whether another developer can run the project, which external accounts it needs, and how configuration changes are reviewed. Do not request private client code the candidate has no authority to share.
Public examples, redacted traces, controlled demonstrations, and specific explanations can still establish expertise. Missing access to confidential material is different from an inability to explain the work.
My Voho case study describes a contribution within an existing team rather than claiming sole ownership. The StackFlight case study describes an independent end-to-end build. Read those categories differently when assessing evidence.
Use a bounded next step
If the portfolio is relevant, commission a small paid review or implementation with clear acceptance cases. A broad unpaid rebuild proposal is a weak substitute for observing how someone diagnoses and communicates a real constraint.
The paid-trial guide provides a useful structure.
Portfolio questions
Does a public demo prove production reliability?
No. Ask what was tested and what operating evidence exists.
Should every strong engineer have revenue metrics?
Not every role owns commercial measurement. Match the claim to the person's actual responsibility.
What should a founder prioritise?
Prioritise relevant workflow evidence, honest scope, and maintainability when discussing a project.