Production AI Systems·Lesson 60 of 67·expert
Retries, timeouts, backoff and rate limits
Retry the right failures, back off with jitter so your clients stop synchronising, respect Retry-After, and shed load before the service does it for you.
In this lesson · 22 min
- →Separate retryable failures from permanent ones
- →Compute exponential backoff, and explain why jitter is not optional
- →Respect a Retry-After header over your own schedule
- →Rate-limit yourself with a token bucket
- →Break the circuit when a dependency is down
A client that retries everything is as broken as one that retries nothing — it just fails more expensively. This lesson is about failing well.
What to retry
Exponential backoff
Each attempt waits longer than the last, so a struggling service gets room to recover:
Jitter
Pure exponential backoff has a failure mode that only appears under load. A hundred clients hit a 429 at the same moment, all wait exactly 0.5s, and all retry at the same moment — recreating the overload they were backing off from:
Without jitter every client retries in the same instant. With it they spread out, and the service sees a trickle rather than a wall.
Retry-After
When the service tells you how long to wait, that beats any schedule you invented:
A complete retry wrapper
Three things there are worth copying: a total time budget as well as an attempt count, honouring
retry_after when the service supplies one, and never retrying a Permanent.
Rate-limiting yourself
Waiting for a 429 is reactive. A token bucket keeps you under the limit in the first place:
The burst capacity absorbs a spike; the refill rate holds the long-run average. Five per second with a capacity of five means “five instantly, then one every 200 ms”.
Circuit breaker
When a dependency is genuinely down, retrying every request wastes time and keeps the pressure on. A breaker fails fast until it has reason to think the service is back:
Three failures open the circuit; further calls are refused without touching the network. After the cooldown one probe is allowed through, and its result decides whether the circuit closes again.
Try it yourself
Write backoff_delays(attempts, rng, base, cap): full jitter — each delay is a uniform draw between 0 and min(cap, base * 2 ** attempt).
Complete TokenBucket: refill by elapsed time up to the capacity, and take a request only when enough tokens are available.
Check yourself
A hundred clients hit a 429 at the same moment and all back off by exactly 0.5 seconds. What happens?
Check yourself
Why run a token bucket when you already retry on 429?
Recap
- Retry 429, 5xx and connection failures. Never retry 400, 401, 404 or 413.
- Exponential backoff with a cap, and full jitter — deterministic backoff keeps a fleet synchronised and prolongs the outage.
Retry-Afterbeats your own schedule.- Budget total time as well as attempts, or five retries with a 60-second cap can block for minutes.
- A token bucket paces you under the limit; model APIs limit on requests and tokens, so run two.
- A circuit breaker fails fast during a real outage and probes once after a cooldown.