Inspect retrieval before replacing the language model when answers are unsupported. A stronger model cannot reliably cite a missing exception or recover a document the pipeline never supplied.
Retrieval comparison isolates the evidence stage
A retrieval evaluation compares candidate passages against reviewed relevance and sufficiency expectations. It should precede answer-generation changes when the failure may originate in ingestion, search, filtering, or context assembly.
For a proposed comparison, pin the corpus snapshot, query set, permissions, and relevant configuration. Record candidate ranks, final context, source versions, and expected evidence. Keep answer evaluation separate so model memory does not conceal a missing retrieval path.
Do not assume every wrong answer is a retrieval defect. If complete appropriate evidence was supplied and the model misinterpreted it, the fix may belong in generation or the response contract. Evidence localisation determines the next change.
Diagnose a fictional policy omission
The assistant gives a general eligibility rule but misses a product-version exception. Review shows that the source parser dropped the footnote during ingestion. Changing the language model would not repair that lost information.
A second fixture includes the footnote in the index but not the final candidate budget. That points toward ranking or context assembly. A third supplies the footnote correctly but the model ignores it, shifting investigation to interpretation.
| Evidence observation | Next boundary |
|---|---|
| Source text missing after ingestion | Extraction and structure |
| Indexed passage never retrieved | Query, filters, or search |
| Relevant candidate omitted from context | Ranking and assembly |
| Complete evidence supplied, answer wrong | Generation and claim rules |
| Source itself contradictory | Content authority and ownership |
Replay the original failed question after a narrow fix and include neighbouring cases. Preserve the initial artifact before experimentation so later runs do not erase the mechanism.
Measure enough to choose a remedy
Track relevant-passage recall within a defined budget, distracting results, condition coverage, and permission failures. Report counts and query classes rather than one unexplained relevance score.
Evaluation labels take reviewer effort and may be ambiguous. Use an explicit rubric and keep uncertain cases visible. Avoid changing search, chunking, corpus, and model simultaneously unless operational urgency requires containment first; otherwise causal interpretation becomes difficult.
This diagnostic process is proposed. The voice integration case study describes relevant retrieval and backend work without claiming benchmark gains. Pair it with evaluation failure triage.
Retrieval-first FAQ
Can a better model compensate for bad retrieval?
Sometimes it may answer from general knowledge, but that does not establish current company-policy support.
Should every model upgrade be delayed for retrieval work?
No. Investigate the actual failure boundary and compare controlled evidence.
What should be saved?
Keep question, corpus and configuration versions, permitted candidates, assembled context, expected passages, and the final claim.