Large document imports should respect source API limits and preserve recovery checkpoints. Adding more workers can increase throttling and delay useful updates if every worker competes for the same account quota.
Import throughput has several bottlenecks
Collection indexing includes enumeration, download, extraction, metadata checks, representation generation, and index writes. Each stage may have independent limits. Measure where work actually waits before increasing concurrency.
For a proposed scheduler, scope request budgets to the provider’s documented account or resource limits. Use supported pagination and change tracking, bounded retries, and durable progress. Keep expensive extraction separate from source-fetch admission where practical.
Do not hard-code a universal request rate from an unrelated service. Current provider documentation and actual account configuration determine the limits and retry guidance that apply.
Simulate a fictional import burst
A connector discovers 20,000 documents. Download workers immediately saturate the source account, while extraction workers remain idle waiting for responses. Doubling all workers can worsen the bottleneck instead of improving time to usable evidence.
A controlled fixture returns quota responses and delayed pages. The scheduler should defer eligible requests, preserve the page or change checkpoint, and continue without skipping resources or repeating accepted effects. The collection size illustrates the workload, not measured capacity.
| Load condition | Required invariant |
|---|---|
| Ordinary pagination | Every supported page processed |
| Rate-limit response | Documented bounded retry behaviour |
| Worker restart | Durable progress resumes safely |
| One very large resource | Other work not starved indefinitely |
| Urgent permission removal | Security update path not trapped behind bulk text work |
Track source requests, queue age, accepted resources, failed stages, and active eligible content. A high fetch rate is not useful if extraction failures leave documents unsearchable.
Separate bulk ingestion from currentness
An initial crawl and ongoing edits have different urgency. Define fairness or priority so new important changes are not indefinitely delayed behind a large historical import. Avoid letting arbitrary document text assign its own priority.
More scheduling control adds complexity and can reduce peak throughput. It buys predictable source load and recoverable progress. Evaluate complete ingestion time, freshness, quota pressure, cost, and access correctness rather than one optimistic documents-per-second figure.
This indexing design is proposed. The voice integration case study describes related backend and cloud work without capacity benchmarks. Pair it with retry-delay design, whose bounded scheduling principles also apply here.
Large-import FAQ
Should every worker retry immediately after throttling?
No. Coordinate under the actual quota scope and supported retry guidance.
Is download completion equivalent to index readiness?
No. Extraction, permissions, representation, and activation still need evidence.
What should a scale test prove?
The pipeline makes durable complete progress under limits and failures while preserving freshness and resource access boundaries.