Timeouts, Retries & Backoff
Every call across a network can fail in ways a local function call cannot: the network can drop packets, a backend can be slow instead of down, and "no response yet" is indistinguishable from "never coming." How a client waits, gives up, and tries again determines whether one slow backend stays a contained blip or becomes a fleet-wide outage.
Connection timeout versus request timeout
These bound two different waits, and confusing them is a common source of outages:
| Connection timeout | Request timeout | |
|---|---|---|
| Bounds | Time to establish the TCP/TLS connection | Time from sending the request to receiving a full response |
| Fires when | The server is unreachable, overloaded at the socket level, or the network is dropping SYNs | The server accepted the connection but is slow to process or respond |
| Typical value | Low - hundreds of ms to a couple seconds | Depends entirely on the operation - reads short, heavy writes/reports longer |
| A high value costs you | Threads/sockets tied up waiting on hosts that are simply gone | Threads tied up waiting on a backend that is alive but struggling |
A connection can succeed in 20ms and the request can still hang for 30 seconds
- they need independent budgets. Every outbound call should set both explicitly; the language/library default is often "wait forever," which turns one wedged dependency into a slow-motion pile-up of exhausted threads upstream.
Retry storms
A single failed request retried once seems harmless. The trouble starts when many clients retry many failures at once, against a backend that is failing because it is overloaded:
- A backend gets slow under load - requests start timing out.
- Every client that timed out retries immediately.
- The retries land on the same overloaded backend, adding load on top of load, at the exact moment it needed less traffic to recover.
- More requests time out, triggering another wave of retries. Each round multiplies the traffic the backend receives.
This is a retry storm (or "retry amplification"): the very mechanism meant to smooth over transient failures turns a partial slowdown into a total outage, because every layer that calls the backend retries independently, without knowing every other caller is doing the same thing at the same moment.
Retries are amplifiers, not just safety nets. If service A calls B calls C, and each layer retries 3 times, one failure at C can turn into up to 9 actual requests reaching C from a single original call. Stack a few layers of "just retry 3 times" and a brief blip at the bottom of the stack becomes an order-of-magnitude traffic spike exactly when the system is least able to absorb it.
Exponential backoff and jitter
The fix is to make retries spread out instead of pile up:
- Exponential backoff - wait longer between each retry: 1s, then 2s, then 4s, then 8s, instead of retrying immediately every time. Failures get fewer and further apart, giving the backend room to recover instead of a constant drumbeat of retries.
- Jitter - add randomness to each wait so retries from different clients don't land in the same instant. Without jitter, every client that failed at the same moment backs off by the same fixed amount and then retries in the same synchronized wave, all over again. A common pattern is "full jitter": pick a random wait between 0 and the backoff ceiling for that attempt, rather than the ceiling itself.
Backoff without jitter still herds; jitter without backoff still floods immediately once. The combination is what actually spreads retries out over time and across clients.
Circuit breakers
Backoff still means every client keeps trying the failing backend, just more slowly. A circuit breaker goes further: after enough failures, it stops sending requests to a backend at all for a while, failing fast locally instead of waiting for a timeout on every call.
| State | Behavior | Transitions to |
|---|---|---|
| Closed | Requests pass through normally; failures are counted | Open, once failures cross a threshold |
| Open | Requests fail immediately without touching the backend at all | Half-open, after a cooldown timer expires |
| Half-open | A small number of test requests are allowed through | Closed if they succeed; back to Open if they fail |
The open state is the key idea: it protects the failing backend from further load, and protects the caller from burning threads and time on calls that would fail anyway - both sides benefit from the breaker refusing to try.
Idempotency: when retries are safe at all
None of this matters if a retried request causes harm. A request is
idempotent if performing it twice has the same effect as performing it
once. GET /orders/42 is naturally idempotent - reading twice changes
nothing. POST /orders to create a new order is not - retrying it blindly can
create two orders for one purchase.
An idempotency key fixes this: the client generates a unique key per logical operation and sends it with the request. The server remembers keys it has already processed and, if the same key arrives again (because the client retried after a timeout, not knowing whether the first attempt succeeded), returns the original result instead of repeating the side effect.
- Retry with backoff when the operation is idempotent and failures are expected to be transient (a dropped packet, a momentary blip) - a couple of retries with backoff and jitter usually recovers cleanly.
- Circuit breaker when a backend is failing persistently - retrying a backend that has been down for the last thirty seconds only adds load and latency; failing fast is strictly better for both sides.
- Both together in production systems: the breaker decides whether to try at all, and backoff-with-jitter governs the spacing of the attempts it does allow through, including the half-open probes.
- Neither for non-idempotent operations without an idempotency key - retrying blindly risks double-charging a card or double-creating a record; fix the idempotency problem before adding retries.
Request hedging
Backoff and breakers react to failure. Hedging reacts to slowness: send the request to one backend, and if no response arrives within some delay (commonly around the median or p95 latency for that call), fire a second, identical request to another backend, then take whichever response comes back first and discard the other.
Hedging trades a small amount of extra load - most hedge requests never fire, because most responses beat the delay - for a large cut in tail latency, because the rare slow instance no longer decides how long the whole call takes. It works best for idempotent reads against a pool of interchangeable backends, for exactly the reason retries need idempotency: firing a second identical request must be harmless.
Recap
- Set connection timeouts and request timeouts independently; neither should default to "forever."
- Naive, synchronized retries amplify load onto an already-struggling backend and can turn a blip into an outage - the retry storm.
- Exponential backoff spaces retries out over time; jitter spreads them across clients so they stop arriving in synchronized waves.
- A circuit breaker stops calling a persistently failing backend (closed → open → half-open → closed), protecting both caller and callee.
- Retries are only safe for idempotent operations; idempotency keys make non-idempotent operations (like creating an order) safely retryable.
- Hedging fires a second request after a delay and takes whichever answers first, cutting tail latency for idempotent reads.