Keyword and vector search fail differently on company documents. Compare them against the questions employees actually ask, especially exact identifiers, informal descriptions, and permission-sensitive material.
Search methods match different signals
Keyword search ranks content using textual features under the engine’s analysis and scoring rules. Vector search compares learned representations of queries and passages. Similarity can help with paraphrases, but it does not establish factual support or source authority.
For a proposed evaluation, create a small approved document collection and reviewed questions with relevant source passages. Include product codes, policy names, abbreviations, natural-language descriptions, and intentionally unanswerable questions.
Elastic’s hybrid-search documentation describes combining full-text and vector results. The existence of that option is a reason to test complementarity, not proof that hybrid retrieval wins on every corpus.
Compare fictional company questions
An employee asks for “XR-17 reset procedure.” Exact product text matters because a semantically similar procedure for XR-71 could be dangerous. Another asks “what do I do when a device keeps restarting?” even though the document says “repeated reboot.” Semantic retrieval may help connect those expressions.
Evaluate whether the correct supporting passage appears within the chosen result budget. Do not grade only the generated answer, because model memory can hide missing retrieval evidence.
| Query class | Retrieval concern |
|---|---|
| Exact product identifier | Preserve discriminating tokens |
| Informal paraphrase | Find differently worded relevant material |
| Shared acronym | Resolve context rather than broad similarity |
| Current policy | Respect version and authority |
| Unsupported question | Avoid treating nearest content as an answer |
Illustratively, retrieving four relevant passages among five returned is 80 percent precision for that labelled query result. It does not measure answer correctness or performance across the whole company. These are explanatory values, not observed benchmark results.
Compare operational cost as well as relevance
Vector ingestion adds embedding and index-management work. Keyword systems need suitable analysis, synonyms, and field design. Both require content updates, permissions, and evaluation ownership.
Start with a measured baseline and add complexity where it resolves specific failures. A retrieval method can improve candidate recall while introducing irrelevant neighbours, so inspect both missed evidence and distracting results under the same corpus and query set.
This evaluation is proposed. The voice integration case study describes retrieval experience without claiming comparative search benchmarks. Pair it with hybrid exact-name search.
Keyword versus vector FAQ
Does vector similarity prove the passage answers the question?
No. Relevance and factual support still need evaluation.
Should exact identifiers use only vectors?
Test literal matching and analysed fields carefully; small token differences can matter materially.
What should choose the method?
Use reviewed retrieval results, permissions, freshness, latency, cost, and maintenance under the actual supported questions.