Skip to content

Production AI Systems·Lesson 60 of 67·expert

Retries, timeouts, backoff and rate limits

Retry the right failures, back off with jitter so your clients stop synchronising, respect Retry-After, and shed load before the service does it for you.


In this lesson · 22 min

  • →Separate retryable failures from permanent ones
  • →Compute exponential backoff, and explain why jitter is not optional
  • →Respect a Retry-After header over your own schedule
  • →Rate-limit yourself with a token bucket
  • →Break the circuit when a dependency is down

A client that retries everything is as broken as one that retries nothing — it just fails more expensively. This lesson is about failing well.

What to retry

PythonPython not started

Exponential backoff

Each attempt waits longer than the last, so a struggling service gets room to recover:

PythonPython not started

Jitter

Pure exponential backoff has a failure mode that only appears under load. A hundred clients hit a 429 at the same moment, all wait exactly 0.5s, and all retry at the same moment — recreating the overload they were backing off from:

PythonPython not started

Without jitter every client retries in the same instant. With it they spread out, and the service sees a trickle rather than a wall.

Retry-After

When the service tells you how long to wait, that beats any schedule you invented:

PythonPython not started

A complete retry wrapper

PythonPython not started

Three things there are worth copying: a total time budget as well as an attempt count, honouring retry_after when the service supplies one, and never retrying a Permanent.

Rate-limiting yourself

Waiting for a 429 is reactive. A token bucket keeps you under the limit in the first place:

PythonPython not started

The burst capacity absorbs a spike; the refill rate holds the long-run average. Five per second with a capacity of five means “five instantly, then one every 200 ms”.

PythonPython not started

Circuit breaker

When a dependency is genuinely down, retrying every request wastes time and keeps the pressure on. A breaker fails fast until it has reason to think the service is back:

PythonPython not started

Three failures open the circuit; further calls are refused without touching the network. After the cooldown one probe is allowed through, and its result decides whether the circuit closes again.

Try it yourself

Exercise 1

Write backoff_delays(attempts, rng, base, cap): full jitter — each delay is a uniform draw between 0 and min(cap, base * 2 ** attempt).

Your turnPython not started
Exercise 2

Complete TokenBucket: refill by elapsed time up to the capacity, and take a request only when enough tokens are available.

Your turnPython not started

Check yourself

A hundred clients hit a 429 at the same moment and all back off by exactly 0.5 seconds. What happens?

Check yourself

Why run a token bucket when you already retry on 429?

Recap

  • Retry 429, 5xx and connection failures. Never retry 400, 401, 404 or 413.
  • Exponential backoff with a cap, and full jitter — deterministic backoff keeps a fleet synchronised and prolongs the outage.
  • Retry-After beats your own schedule.
  • Budget total time as well as attempts, or five retries with a 60-second cap can block for minutes.
  • A token bucket paces you under the limit; model APIs limit on requests and tokens, so run two.
  • A circuit breaker fails fast during a real outage and probes once after a cooldown.