The Retry That Took Down the System.

By Muhammad Umar·January 28, 2026·8 min read·Issue #13

The outage does not start with anything failing. It starts with a database that usually answers in 20 milliseconds taking 400.

Nothing is broken. Nothing alerts. Every request still succeeds.

Except that 400 milliseconds is past somebody’s timeout. So a service gives up waiting and tries again, which is exactly what you built it to do, and which is the correct behaviour that prevents your customer being charged nothing at all.

Ninety seconds later everything is down, and the database that was merely slow is now refusing connections entirely.

Nothing failed and then caused an outage. The recovery mechanism caused the outage.

The problem: retries add load exactly when you have none to spare

A retry is not free. It is another request, sent to a component that has just demonstrated it is struggling to serve the requests it already has.

Under normal conditions this costs nothing, because retries almost never fire. That is the trap. Retry logic is dormant while everything is healthy and activates in unison the moment anything degrades, so the extra load arrives precisely when there is no headroom for it.

A system can be running comfortably at 40% capacity and still die from this, because the load it is designed for is not the load it will receive during an incident.

Retries multiply, they do not add

The part that turns a wobble into an outage is that retries usually exist at several layers, and each layer is unaware of the others.

Three tiers each retrying three times is not nine attempts. It is twenty-seven, because the middle tier retries each of the attempts made above it.

And the multiplication lands entirely on whichever component is already the slowest, since that is the one causing the timeouts in the first place. The struggling database receives 27 times its normal traffic at the exact moment it is least able to serve it.

Every layer you add is another multiplier, and nobody owns the total because each layer is being individually reasonable.

Everyone retries at the same instant

There is a second effect that turns a recoverable incident into an oscillating one.

Say a service is briefly unavailable and a thousand clients all fail. If they each wait one second and try again, they do not spread out. They arrive together, in a spike, one second later.

The failure synchronised them. Before the incident their requests were spread across time by ordinary randomness. The outage lined them all up, and every subsequent round keeps them lined up.

This is why some outages have a rhythm to it. The service comes back, gets flattened by the synchronised herd, fails again, and the whole thing repeats on a cycle you can see in the graphs.

The fixes, in the order they are worth doing

Back off exponentially

Waiting a fixed interval between attempts keeps the pressure constant. Doubling the wait after each failure means a client that keeps failing withdraws from the system rather than hammering it.

// Wait 100ms, then 200, 400, 800, capped at 10 seconds.
function delayFor(attempt) {
  return Math.min(100 * 2 ** attempt, 10_000)
}

This alone is a large improvement, and it is where most teams stop. It is also not enough on its own.

Add jitter, which is the part people skip

Exponential backoff spreads a single client’s attempts over time. It does nothing about a thousand clients, because they all compute the same delay and remain synchronised. You have simply moved the spike further out.

The fix is to randomise each wait.

// Full jitter: pick a random point anywhere in the backoff window.
function delayFor(attempt) {
  const ceiling = Math.min(100 * 2 ** attempt, 10_000)
  return Math.random() * ceiling
}

AWS published simulations of these strategies, and the version that picks a random value across the whole window rather than adding a small wobble to a fixed one performed best on both total work done and time to completion.¹

It is one line, it is easy to leave out, and leaving it out is what preserves the herd.

Retry at one layer only

The multiplication problem has no clever solution. It has an organisational one: decide which layer owns retrying, and remove it everywhere else.

Usually that layer is the one closest to the user, because it is the only one that knows whether the whole operation is still worth completing. The database client does not know that the person who made the request closed the tab forty seconds ago.

This is harder than it sounds, because retries arrive by default. Your HTTP client library has them on, your database driver has them on, your service mesh has them on, and your job runner has them on. Most amplification is not a decision anybody made.

Give retries a budget

Backoff limits how often one client retries. It does not limit the total. If enough clients are failing, a system can still be overwhelmed by attempts that are individually well behaved.

A retry budget caps retries as a proportion of normal traffic, commonly around 10%. Once the budget is exhausted, failures are returned immediately rather than retried. Google describes this approach and the related client-side throttling in its site reliability material.²

The reasoning is worth stating plainly. When most requests are failing, retrying is pointless anyway. A budget converts an unbounded multiplier into a bounded one, which is the difference between a slow patch and an outage.

Stop calling a dependency that is down

A circuit breaker watches the failure rate and, once it crosses a threshold, stops sending requests entirely for a period. It then lets a small number through to check whether the dependency has recovered.³

This does two useful things at once. It removes load from something that is already failing, giving it room to recover, and it stops your service from tying up its own threads and connections waiting for calls that will time out.

That second effect is what stops a failure spreading sideways. Without it, one slow dependency exhausts your connection pool, and now your service is unavailable for requests that never needed that dependency at all.

Do not retry things that cannot succeed

The cheapest fix on this list, and the most commonly missing.

A malformed request will be malformed on the second attempt. An authorisation failure will still be unauthorised. Retrying a client error is pure amplification with no possible upside, and it is what a naive catch-and-retry wrapped around everything produces.

// Retry only what a later attempt could plausibly fix.
function shouldRetry(response) {
  if (response.status === 429) return true       // rate limited, back off
  if (response.status >= 500) return true       // server side, may recover
  return false                                   // 4xx will fail identically
}

Why this never shows up in testing

It is worth being explicit about why competent teams ship this, because the answer is not carelessness.

Retry paths are dormant during every normal test. Your load tests exercise the healthy path at high volume, and your integration tests exercise correctness at low volume. Neither produces the condition that matters, which is a dependency that is slow rather than down, under real traffic.

A dependency that is down fails fast and often causes no amplification at all, because connections are refused immediately. A dependency that is merely slow holds every caller open until they time out, which is what fills the queues and triggers the retries.

So the test that finds this is not a load test. It is deliberately adding latency to a dependency under production-like traffic and watching what your own system does in response.

When this is the wrong advice

When you have one client and a small system. A background job that calls one API on a schedule can retry naively forever without consequence. Circuit breakers and budgets are machinery for scale you may not have, and adding them early gives you components to tune and no failures to prevent.

When failing fast is worse than the load. A circuit breaker turns partial degradation into a definite refusal. For a checkout, refusing immediately may cost more than a slow response would have. Breakers are right when a dependency is genuinely unavailable and questionable when it is intermittently slow but succeeding.

When the operation is not safe to repeat. Everything here assumes retrying is harmless, which it only is if the receiving end deduplicates. Adding aggressive retries to an endpoint with no idempotency key is not resilience engineering, it is a plan for charging people twice at higher volume.

The takeaway

Retries are the right answer to an unreliable network and they are also load, and load arriving during an incident is worth several times load arriving on a quiet afternoon.

So the question to ask about your own system is not whether it retries. It is: if everything got slow at once, how many requests would that turn into? Multiply it out through every layer. If nobody can answer, that number is your real capacity limit, and it is a lot lower than the one on your dashboard.