Retry delays should respect the downstream API’s limits and the business operation’s deadline. Immediate repeated webhook processing can turn a temporary rate limit into a larger backlog and duplicate side effects.
Retry scheduling needs shared context
A rate-limited retry is a deferred attempt after a service refuses or delays work because a quota is exceeded. The policy should distinguish that condition from malformed input, expired credentials, and unknown write outcomes.
For a proposed scheduler, record the logical operation, attempt count, next eligible time, total deadline, and relevant account scope. Use the API’s documented retry guidance where available, then apply bounded backoff and jitter as appropriate.
Do not let each worker independently consume the same shared account quota. Coordinate concurrency and retry allowance at the scope the provider actually limits. Nested SDK and application retries can multiply attempts unexpectedly.
Simulate a fictional rate-limited burst
A hundred customer-update events arrive together, but the destination accepts only a smaller sustained workload under the account’s actual plan. Workers receive rate-limit responses. The scheduler defers eligible work instead of looping immediately or marking every event permanently failed.
If a write response is lost, first determine whether execution may have occurred. Rate-limit recovery does not remove the need for idempotency or reconciliation. A fresh operation identifier on every attempt can create duplicates when the service recovers.
| Failure class | Scheduling decision |
|---|---|
| Documented rate limit | Respect supported retry guidance and quota scope |
| Invalid payload | Repair or reject; do not repeat unchanged |
| Credentials rejected | Route to configuration recovery |
| Write status unknown | Reconcile or use supported duplicate protection |
| Deadline exceeded | Create an owned unresolved disposition |
Test with a fake API returning controlled responses and record simulated time advances. This produces a reproducible schedule without waiting through real delays or consuming live quota.
Preserve fairness and business priority
A large tenant or noisy event type should not starve unrelated work. Define queue priority and fairness explicitly. High-priority recovery may need reserved capacity, but priority labels should come from trusted policy rather than arbitrary event content.
Longer delays reduce pressure but increase data freshness lag. Shorter delays recover faster when limits clear but can waste requests. Show queue age and overdue operations so the tradeoff remains operationally visible.
This retry design is proposed. The voice platform case study describes related integration work. See safe retry side effects and outage recovery.
Retry delay FAQ
Is one fixed delay suitable for every API?
No. Use the actual service contract, quota scope, workload, and business deadline.
Should rate-limited work be retried forever?
No. Define a bounded policy and an accountable recovery route.
What should be monitored?
Queue age, attempts, quota responses, deadline breaches, and final authoritative outcomes.