A small knowledge-base pilot should use documents that can answer a defined set of valuable questions. Starting with every company file makes relevance, permissions, freshness, and ownership harder to diagnose.
Pilot size should follow coverage
A pilot knowledge base is a bounded approved collection with known owners and expected questions. Small means operationally reviewable, not an arbitrary document count. Ten complex policy manuals can be harder to validate than many short articles.
For a proposed selection sheet, record document purpose, authority, audience, effective status, extraction quality, owner, and the questions it should support. Exclude stale drafts and unresolved access problems from the initial supported scope.
Do not select only easy documents to create an impressive demonstration. Include representative tables, conditions, exact identifiers, and a few intentionally unanswerable questions that test the system’s limits.
Build a fictional export-support corpus
A product team pilots answers about export setup and common failures. It chooses approved guides, relevant release notes, and current limitations. Internal incident notes remain outside the public scope unless explicitly authorised for that audience.
The reviewed query set includes a routine setup question, an exact error code, a version-specific limitation, an exception, and a question the corpus cannot answer. The final case should trigger a useful qualified response rather than a plausible invented policy.
| Selection criterion | Evidence to review |
|---|---|
| Valuable question coverage | Approved expected answers and passages |
| Authority | Named owner and publication status |
| Access | Supported audience and resource permissions |
| Extraction | Text and structural fidelity |
| Maintenance | Update and retirement responsibility |
Keep the corpus snapshot and query expectations versioned. Otherwise a changed source can make a later result look like a model regression when the expected policy actually moved.
Measure the whole evidence path
Inspect ingestion, candidate retrieval, context sufficiency, final claims, and source previews. A good answer can come from model memory despite a broken index, so do not use answer quality alone to validate retrieval.
A narrow corpus limits the questions the assistant can support. Make that scope visible and collect approved content-gap feedback. Expansion should add reviewed sources and fixtures deliberately, not silently turn a controlled pilot into a company-wide search promise.
This pilot design is proposed. The voice integration case study describes related retrieval work without claiming this corpus or results. Pair it with knowledge-assistant abstention.
Small knowledge pilot FAQ
Is there one ideal document count?
No. Choose a collection whose authority, structure, access, and question coverage can be reviewed meaningfully.
Should unanswerable questions be included?
Yes. They test whether the assistant respects the evidence boundary.
What should justify adding more sources?
A defined user need, approved ownership and permissions, usable extraction, and evaluation cases for the expanded scope.