Restarting an agent server restores capacity for new work. It does not prove an interrupted caller resumed or that their pending booking completed. Recovery needs a customer-action ledger outside the lost process.
LiveKit's custom deployment guide distinguishes isolated job crashes from an agent-server crash that terminates child jobs. Design against the failure actually being tested, rather than treating every restart as graceful draining.
Persist the evidence required to recover
Before a consequential remote operation, record client scope, request identity, authorized intent, desired effect, and current state. Save the destination identifier and verified outcome when available. Avoid using a final transcript save as the only durable request record.
Define recovery categories: never submitted, explicitly rejected, confirmed remotely, and unknown. Unknown outcomes require reconciliation because a successful remote write may have happened before the process disappeared.
Rehearse a fictional calendar interruption
A caller accepts a visit. The worker submits creation, then the hosting instance fails before it stores the response. The durable record identifies the intended appointment but lacks a confirmed remote result.
The recovery process searches through a supported reference or routes to staff review. It must not automatically create another appointment merely because the new worker has no in-memory history. Any customer callback follows the business's approved communication policy.
| Restart fixture | Required recovery evidence |
|---|---|
| Failure before durable intent | No assumed action completion |
| Persisted but unsubmitted | Approved retry or cancellation |
| Remote write in flight | Uncertain outcome and reconciliation |
| Remote ID already saved | Existing verified outcome retained |
| Customer calls again | Link to pending request where appropriate |
Separate technical recovery from customer resolution
Monitor lost-session signals and unresolved request age. An engineer can restore a worker while operations still needs to contact customers, confirm preferences, or resolve expired times. Assign those owners explicitly.
Durable state adds writes and maintenance, but it enables inspection after abrupt failures. Purely in-process recovery may be simpler for prototypes and unsuitable for consequential actions. A small request ledger can be enough when its states and operators are well defined.
Test abrupt termination independently from planned rollout and record the exact injected failure point. Review caller-disconnect lifecycle for the related distinction between session ending and business completion.
Questions about worker recovery
Does a restart reconnect the same conversation automatically?
Do not assume transparent continuation. Verify the actual SDK, transport, and application behavior, and provide an explicit recovery policy.
Should every pending action be replayed?
No. Reconcile unknown remote outcomes and review requests whose customer context has expired.
What proves recovery is complete?
Service capacity is restored, remote actions are reconciled, and every affected customer request is resolved or has a current owner and next step.