Deduplicating imported documents should reduce redundant context without erasing permissions or meaningful version differences. Similar content is a clue to review, not automatic proof that two resources are interchangeable.
Document deduplication has several levels
Exact-byte duplication, text duplication, near-duplicate passages, verified mirrors, and related policy versions are different relationships. They can require different handling in storage, ranking, context assembly, and audit.
For a proposed importer, preserve original source identities and permissions even when a shared content representation is used internally. Record canonical or mirror relationships only through a supported reviewed process.
Do not merge two client documents solely because their standard wording matches. Their customer relationship and access scope may differ even when most text is identical.
Evaluate a fictional policy imported twice
An approved guide appears in a document store and a mirrored wiki. Both connectors import it. The final answer context should not spend most of its budget on repeated paragraphs while omitting the relevant exception.
A near-duplicate copy changes one product identifier and a negation. The evaluation must preserve that distinction. A broad similarity threshold that collapses it into the original would create an incorrect business rule.
| Duplicate class | Required decision |
|---|---|
| Verified identical mirror | Reduce repeated context with provenance retained |
| Old and current version | Apply authority and effective-status rules |
| Different client, similar text | Preserve resource and permission separation |
| Overlapping chunks | Avoid redundant passages without losing conditions |
| Uncertain relationship | Review rather than destructive merge |
Use a fictional corpus containing exact copies, consequential near-copies, and distinct resources with similar titles. Assert the final evidence set and untouched access boundaries.
Keep deletion semantics safe
If one mirror is removed, decide whether the remaining source still legitimately supports the content. Deleting a shared representation without relationship checks can break valid resources, while keeping it broadly available can retain revoked evidence.
Deduplication can save processing and context budget, but relationship tracking adds complexity. Start with reliable exact or approved mirror cases, then evaluate broader semantic grouping against known failure fixtures. Do not claim storage savings or relevance gains before measuring them.
This importer design is proposed. The voice integration case study describes related retrieval work without claiming deduplication benchmarks. Pair it with repeated-context reduction.
Add a removal fixture for one of two authorised mirrors and verify that remaining valid evidence stays available only through its own current permissions.
Imported duplicate FAQ
Does identical text imply identical permission?
No. Preserve each source resource’s access relationship.
Should deduplication delete source history?
Keep the provenance and version relationships needed for review and supported recovery.
What proves useful deduplication?
Redundant context decreases while necessary conditions, authority, source lineage, and resource boundaries remain correct.