Ingestion delay should measure the time from a meaningful source change to usable current evidence. A fast download metric can hide a long extraction queue or an index that has not activated.
Delay needs explicit start and end points
Knowledge-ingestion delay can describe source-to-observation, observation-to-processing, processing-to-index, or end-to-end currentness. Record these stages separately so the metric identifies the bottleneck.
For a proposed event model, preserve source identity and version, source change time where trustworthy, connector observation, durable acceptance, processing completion, and active eligibility. Mark unknown source timestamps instead of inventing precision.
Do not merge content and permission delay into one harmless average. A delayed typo correction and a delayed access revocation can have very different consequences and response requirements.
Calculate a fictional ingestion timeline
A policy changes at 10:00, is observed at 10:04, finishes extraction at 10:06, and becomes query-eligible at 10:09. End-to-end delay is nine minutes, with four minutes before observation and five afterward.
If a generated-answer cache continues serving the old result until 10:15, user-visible freshness may lag further. That is a separate derived-path measurement. These times illustrate definitions, not measured guarantees.
| Timing boundary | Diagnostic question |
|---|---|
| Source to observation | Is polling or notification late? |
| Observation to worker start | Is there a queue backlog? |
| Processing duration | Is extraction or representation slow? |
| Index completion to activation | Is serving state delayed? |
| Activation to current answer | Are caches or sessions stale? |
Test clock differences and missing metadata. A negative calculated delay can reveal incompatible clocks rather than magical fast processing.
Show distributions and unresolved work
Averages can hide a few very old documents. Report counts, selected percentiles where justified, oldest pending age, and failure categories. Include resources that never complete rather than measuring only successful imports.
Detailed timing adds telemetry and privacy considerations. Use opaque references and safe categories, then provide authorised drill-down. Choose alert thresholds from the actual business freshness requirement and staffing response, not a universal number.
This metric design is proposed. The voice integration case study describes related knowledge work without latency benchmarks. Pair it with answer freshness display.
Include one permanently failed document and one late permission event in the fixture. Both must remain visible with their own operational meaning rather than disappearing from successful-import statistics.
Ingestion delay FAQ
Is processing duration the same as end-to-end delay?
No. Observation, queueing, activation, and cache stages can add time.
Should failed imports be excluded?
Keep their unresolved age and count visible so the metric does not ignore the worst cases.
What should determine a freshness target?
Use the consequence of stale evidence, supported source behaviour, architecture, and accountable response policy.