A developer can build a convincing document chatbot while overlooking the rule that one customer must never retrieve another customer's material. Hiring for knowledge retrieval requires checking source access and document lifecycle behaviour alongside answer quality.
This is a practical interview plan for founders commissioning an internal knowledge assistant or customer-facing document product. Use a small fictional document set so the candidate can explain decisions without accessing confidential company files.
What does a knowledge retrieval engineer need to understand?
Retrieval engineering connects ingestion, document structure, search, permissions and answer generation. The engineer should explain how a source enters the system, becomes searchable, changes over time and eventually leaves it.
Ask where access is checked and which identities are trusted. A tenant identifier supplied by a browser is a claim that the server must verify, not sufficient authority to search that tenant's documents.
Give the candidate a document lifecycle scenario
Prepare two fictional teams and three short policies. One policy has an updated version; another is visible only to one team. Include a question the documents cannot answer.
| Interview task | Useful explanation | Weak shortcut |
|---|---|---|
| Answer an access-limited question | Retrieval applies verified permissions | Hide the citation after generation |
| Update an old policy | Identify replacement and indexing delay | Leave both versions without precedence |
| Delete a source | Remove searchable and cached derivatives | Delete only the original upload |
| Answer an unsupported question | State the evidence gap | Guess from unrelated passages |
The candidate does not need to implement a complete platform during an interview. Ask them to draw the data flow and identify where failures would become visible to an operator.
Review evaluation design before choosing a database
A good answer explains how to assess the retrieved passages separately from the final response. If the required source was never returned, changing the response prompt may conceal the symptom without fixing retrieval.
Ask for examples of expected answers, acceptable source passages and refusal conditions. Include difficult cases involving stale versions, ambiguous terminology and missing access. A single aggregate score makes it hard to decide which subsystem needs work.
My knowledge platform contribution provides relevant project context, but any new product still needs its own document fixtures and acceptance criteria. The source mix and permission rules determine much of the engineering work.
Choose depth that matches the actual integration
A small curated handbook may not need a complex connector system. A business relying on changing permissions across several document services needs explicit sync and recovery design. Hiring a general application developer can work when those boundaries are narrow and reviewed by someone who understands the risks.
Write the first milestone around one complete document lifecycle, including a permission change and deletion. This gives you evidence about the system that a polished answer demonstration cannot provide.
Does vector database experience prove retrieval expertise?
It is one relevant skill, but it does not establish permission handling, source freshness or evaluation quality.
Should the interview use our private documents?
Use fictional fixtures first and grant limited real access only when the engagement requires it.
What should a retrieval pilot deliver?
A working lifecycle, reviewable evidence and known limitations belong in the external engineering brief.