1. What is the Retry pattern, and why do we need it?
The Retry pattern lets an application handle anticipated, temporary failures when it talks to a remote service or resource, by transparently repeating the failed operation — usually after a short wait. Microsoft's Azure Architecture Center lists it as a core cloud design pattern, and it supports the Reliability pillar of the Azure Well-Architected Framework.
Everyday analogy: you call a friend and hear “the number is busy”. You don't give up forever, and you don't redial 50 times a second. You wait a little, try again, wait a bit longer, and after a few tries you send a message instead. That is retry with backoff, a retry limit and a fallback.
In the cloud, failures are normal rather than exceptional. Your code runs on shared hardware, behind load balancers, talking to services that scale, restart, fail over and throttle. Many of those failures fix themselves within milliseconds or seconds. Without retries, every one of them becomes an error page; with naive retries, they can become an outage. The rest of this guide is about the difference.
Retry helps when…
the failure is transient (likely to succeed if repeated soon) and the operation is safe to repeat.
Retry hurts when…
the failure is permanent, the service is overloaded, or repeating the call causes side effects like double charges.
Golden rule
Retry few times, wait longer each time, add randomness, stop at a time budget, and retry at one layer only.
2. Transient vs. non-transient faults
A transient fault is a temporary failure that is likely to disappear on its own: a momentary loss of network connectivity, a service that is briefly unavailable while it restarts, or a timeout because a service is busy. The entire pattern depends on telling these apart from failures that will happen again no matter how many times you try.
RETRYUsually transient
HttpRequestException— connection reset, refused, DNS blip- 408 Request Timeout
- 429 Too Many Requests (throttling — respect
Retry-After) - 500, 502, 503, 504 server-side / gateway errors
- Polly
TimeoutRejectedException(an attempt took too long) - SQL failover, deadlock victim, “server busy” errors (
DbException.IsTransient) - Cloud throttling (Cosmos DB 429, Storage “ServerBusy”)
DON'T RETRYNot transient
- 400 Bad Request, 422 validation errors — the input is wrong
- 401 / 403 — refresh credentials first, then maybe retry once
- 404 Not Found (except brief read-after-write lag)
- 409 / 412 conflicts — re-read and redo the business step instead
- Business-rule errors (“insufficient funds”)
- Bugs:
NullReferenceException,ArgumentException OperationCanceledException— the caller gave up; respect it
Status codes aren't the whole story. A 503 that lasts three seconds is transient; a 503 that lasts thirty minutes is an outage. That is why retries are paired with a limit, a total time budget and a circuit breaker that notices long-lasting failure and stops sending traffic.
3. How a retry works, step by step
When an operation fails, the application has three basic choices (straight from the Azure pattern):
Cancel
The fault isn't transient or won't succeed if repeated. Stop and report an exception.
Retry immediately
The fault is rare and unusual, such as a corrupted network packet. Try once more right away.
Retry after a delay
The fault is a common connectivity or “busy” failure. Wait for the backlog to clear, then retry — waiting longer each time.
The retry loop
Four decisions define every retry policy you will ever write:
- What to retry — which exceptions and results count as transient (
ShouldHandle). - How many times — a hard cap (
MaxRetryAttempts) plus an overall time budget. - How long to wait — the backoff curve (
BackoffType,Delay,MaxDelay), randomised with jitter, or dictated by the server'sRetry-After. - What happens at the end — rethrow, return the last result, fall back, or log and alert.
4. Retry simulator
Pick a failure type and a policy, then press Send request. Watch each attempt travel to the service, see the wait between attempts, and read the log. The C# below the simulator updates to match your settings.
Delays use the same formulas as Polly v8: constant and linear jitter add ±25%; exponential jitter uses Polly's decorrelated-jitter formula. Each attempt in the simulation takes 0.4 s.
5. Backoff strategies: how long to wait
Backoff is the delay between attempts. Waiting gives the dependency time to recover; waiting longer each time reduces pressure on a service that is still struggling. Polly v8 offers three curves through DelayBackoffType:
| BackoffType | Delay before retry n (n = 0, 1, 2…) | Delay = 1 s gives | Use for |
|---|---|---|---|
| Constant (Polly's default) | Delay | 1, 1, 1, 1 s | Quick, rare glitches; tests |
| Linear | Delay × (n + 1) | 1, 2, 3, 4 s | Gentle growth when the recovery time is predictable |
| Exponential | Delay × 2ⁿ | 1, 2, 4, 8, 16 s | Most remote calls — the industry default |
MaxDelay caps any single wait, which matters for exponential growth (the 10th exponential retry at 1 s base would wait 8.5 minutes). If you supply a DelayGenerator, its value wins — unless it returns null or a negative TimeSpan, in which case Polly falls back to the normal calculation. Note that MaxDelay is not applied to generated delays.
Choose by caller, not by habit. An interactive request (a user is waiting) should fail fast: 1–3 retries, short delays, small total timeout, then show a friendly message. A background job can afford many retries with exponential growth and a large MaxDelay.
6. Jitter and the retry storm (thundering herd)
Here is the trap. When a service has an outage, every client fails at the same moment. If they all use the same backoff, they all retry at the same moment too — 0.5 s later, 1.5 s later, 3.5 s later… Each synchronised wave can be bigger than the service can handle, so the recovering service is knocked over again. That is a retry storm.
Jitter adds randomness to each delay so the retries spread out over time. In Polly, UseJitter = true adds a random ±25% for constant and linear backoff, and switches exponential backoff to a decorrelated jitter formula. The standard HTTP resilience handler turns jitter on by default.
Simulation: 400 clients send a request during a 1.5 s outage. Afterwards the service can serve 50 requests per 100 ms (dashed line); anything above that fails as overload. Each client retries up to 6 times with a 0.5 s base delay (capped at 10 s).
Retry budgets
A retry budget (or retry quota) limits how many retries a client may make relative to its recent success. The AWS SDKs, for example, keep a per-client token bucket of 500 tokens: each retry spends tokens and successful calls slowly refill it, and when the bucket is empty the SDK fails immediately instead of retrying. Budgets deliberately trade some extra fast failures for protecting a struggling dependency. In .NET you can build the same idea with a ShouldHandle predicate that checks a shared counter, or combine retries with Polly's rate limiter and circuit breaker.
7. Retry amplification in layered systems
Retries multiply. If a gateway retries 3 times, calls a service that retries 3 times, which calls a database client that retries 3 times, a single user click can become 4 × 4 × 4 = 64 calls to the database — exactly when the database is struggling. The Azure guidance is to implement retry logic only where the full context of the failing operation is understood, and to let lower layers fail fast and report back.
Worst-case calls reaching the bottom layer for one request: 64
Watch for hidden retries. Many libraries already retry: Azure SDK clients, the Cosmos DB SDK, Service Bus, EF Core's execution strategy, and the standard HTTP resilience handler that .NET Aspire service defaults add to every HttpClient. Adding Polly on top multiplies the attempts. Pick one layer to own retries.
8. Idempotency: is it safe to repeat?
An operation is idempotent if doing it twice has the same effect as doing it once. Idempotent operations are inherently safe to retry. Others are dangerous, because the service might receive the request, process it successfully, and then fail to send the response. Your retry logic sees a timeout and sends it again.
| HTTP method | Idempotent? | Retry by default? |
|---|---|---|
| GET, HEAD, OPTIONS | Yes (safe, read-only) | Yes |
| PUT, DELETE | Yes by HTTP semantics, if the server implements them correctly | Usually, with care |
| POST, PATCH | No | Only with an idempotency key |
Default to know about: the .NET standard resilience handler retries all HTTP methods by default. Call options.Retry.DisableForUnsafeHttpMethods() to stop automatic retries for POST, PATCH, PUT, DELETE and CONNECT, or DisableFor(HttpMethod.Post, …) for a specific list.
Try it: the lost response
The customer clicks Pay. The payment service charges the card, but the response is lost on the way back, so the client retries.
Making non-idempotent operations safe
- Idempotency key: the client generates a unique key once per business operation (not per attempt) and sends it on every retry. The server stores the result per key and returns the stored result for duplicates.
- Natural keys and upserts: “create order 1234 if it doesn't exist” instead of “create a new order”.
- Idempotent consumer: for queues, record processed message IDs so redelivered messages are ignored.
- Optimistic concurrency: use ETags (
If-Match) so a repeated update doesn't overwrite newer data. - Transactions: when a step in a database transaction fails transiently, retry the whole transaction, not one statement.
9. Retry with Polly v8 (resilience pipelines)
Polly is the standard .NET resilience library and a .NET Foundation project. Version 8 replaced v7's “policies” with resilience pipelines built from strategies, with native async support and built-in telemetry. Retry is a reactive strategy: it re-executes the same callback when it fails, waits between attempts, and stops when it hits MaxRetryAttempts or sees an outcome it doesn't handle.
Your first retry pipeline
RetryStrategyOptions defaults
| Property | Default | What it does |
|---|---|---|
ShouldHandle | Any exception except OperationCanceledException | Decides which exceptions/results are retried. Narrow this — the default retries bugs too. |
MaxRetryAttempts | 3 | Retries in addition to the original call. int.MaxValue = retry forever (rarely wise). |
BackoffType | Constant | Constant, Linear or Exponential. |
Delay | 2 seconds | The base delay. TimeSpan.Zero = immediate retries. |
MaxDelay | null | Caps any calculated delay. |
UseJitter | false | Adds randomness to delays. Turn it on for anything with many clients. |
DelayGenerator | null | Calculate the delay at runtime (e.g. from Retry-After). Return null to use the normal calculation. |
OnRetry | null | Called before each wait — not before the original attempt. Use for logging/metrics. |
If every retry fails, the strategy rethrows the final exception to your code. For result-based pipelines (ResiliencePipeline<T>), the last handled result (for example, the final 503 response) is returned to you.
Handling results, not just exceptions
HttpClient doesn't throw for 503 or 429 — it returns a response. Use the generic pipeline and HandleResult:
Respecting Retry-After
When a server throttles you with 429 or 503, it often says how long to wait in the Retry-After header (seconds or an HTTP date). Waiting less is pointless; waiting with your own exponential curve may be too short or too long.
Logging every retry
Registering pipelines with dependency injection
Build pipelines once and reuse them — they are thread-safe. With Polly.Extensions, register named pipelines and resolve them through ResiliencePipelineProvider<TKey>; telemetry is wired up automatically.
Two speeds: quick retries, then slow retries
Instead of one complicated DelayGenerator, Polly's docs recommend stacking two retry strategies — for example, a few quick jittered retries for blips, wrapped by a few slow retries for longer hiccups.
Long-running background work
Coming from Polly v7? Policy.Handle<T>().WaitAndRetryAsync(…) becomes new ResiliencePipelineBuilder().AddRetry(new RetryStrategyOptions { … }), and PolicyWrap becomes the order in which you add strategies. The v7 API remains available in the Polly package, so you can migrate gradually and then switch to Polly.Core. For HttpClient, Microsoft.Extensions.Http.Resilience succeeds Microsoft.Extensions.Http.Polly.
10. HttpClient resilience: Microsoft.Extensions.Http.Resilience
For HTTP calls, Microsoft ships Microsoft.Extensions.Http.Resilience (10.x on NuGet), built on Polly v8 and integrated with IHttpClientFactory, DI, logging and metrics. One line gives you a production-ready pipeline:
What AddStandardResilienceHandler configures
Five strategies, from outermost to innermost:
| # | Strategy | Default | Why it's there |
|---|---|---|---|
| 1 | Rate limiter | 1,000 concurrent permits, queue 0 | Caps concurrent requests to the dependency. |
| 2 | Total request timeout | 30 s | The whole operation, including all retries, can't exceed this. |
| 3 | Retry | 3 retries, exponential, jitter on, 2 s base delay | Retries transient failures. |
| 4 | Circuit breaker | 10% failure ratio, min. throughput 100, 30 s sampling, 5 s break | Stops calling a dependency that is clearly failing. |
| 5 | Attempt timeout | 10 s | Limits each individual attempt. |
The retry and circuit breaker handle HTTP 500 and above, 408, 429, plus HttpRequestException and TimeoutRejectedException. The retry also honours the Retry-After header by default (ShouldRetryAfterHeader = true).
Do the timeout maths. Three retries with exponential backoff from 2 s wait roughly 2 + 4 + 8 = 14 s, plus up to four 10 s attempts. The 30 s total timeout ends the operation before the worst case, which is intentional: the total timeout is your latency budget. Configure them together.
Customising the standard handler
Defaults for every client, exceptions for some
Add only one resilience handler per client — stacking handlers multiplies retries. To give one client different behaviour, remove the default handlers first:
A fully custom HTTP pipeline
Configuration from appsettings.json
Retry settings are operational knobs. Bind them to configuration so you can tune them per environment; the AddResilienceHandler overload that exposes a ResilienceHandlerContext can also reload them at runtime with EnableReloads.
Hedging, the cousin of retry. AddStandardHedgingHandler() doesn't wait for a failure: if a request is slow, it sends another one in parallel (optionally to a different endpoint) and takes whichever answers first. Use it for latency-critical, idempotent reads.
11. Combining retry with timeout and circuit breaker
Retry alone is not a resilience strategy. In a pipeline, strategies wrap each other in the order you add them: the first one added is the outermost. The proven order is the one the standard handler uses:
Outer → inner
- Total timeout — latency budget for the whole operation
- Retry — repeats the inner part
- Circuit breaker — counts every attempt; opens when the dependency is clearly down
- Attempt timeout — stops one hung attempt so the retry can fire
Why this order
- A per-attempt timeout inside retry turns a hung call into a fast, retryable
TimeoutRejectedException. Without it, one stuck call blocks for minutes and the retry never fires. - A circuit breaker inside retry sees each attempt, so it notices a failing dependency quickly.
- The total timeout outside retry guarantees the user isn't kept waiting through every retry.
Timeout exception type: Polly throws TimeoutRejectedException, not System.TimeoutException. If you write your own ShouldHandle, decide explicitly whether it handles TimeoutRejectedException, and usually don't retry BrokenCircuitException — the circuit is open precisely to stop calls.
Separate failure domains
A pipeline should address one kind of failure. Don't wrap “download the JSON” and “deserialize the JSON” in the same retry: network errors are transient, a malformed payload is not. Retry the network call; handle parsing separately.
12. Logging, telemetry and alerting
Retries hide problems — that is their job — so you must make them visible to operators.
- Log levels: log early failures that were later retried successfully as informational/warning, and only the final failure as an error. Otherwise operators are flooded with alerts for problems that fixed themselves.
- Polly telemetry: the retry strategy reports an
ExecutionAttemptevent for every attempt (Information when no retries happened, Warning when retries were performed, Error for the final unsuccessful attempt) and anOnRetryevent (Warning) before each retry.Attempt: '0'is the original call. - Metrics to watch: retry rate per dependency, attempts per successful operation, final-failure rate, time spent waiting, circuit-breaker state changes.
- Alert on trends: a rising retry rate is an early warning that a dependency is degrading or needs to be scaled out.
- Correlate: keep the same trace ID across attempts so a single user request shows all its retries in your distributed traces.
13. Testing retry logic
Retry code only runs when things go wrong, so it is easy to ship untested. Test it against a variety of failure conditions and check it doesn't overload services or create race conditions.
- Unit tests: count attempts with a fake that fails N times, and use
Delay = TimeSpan.Zero(or a fakeTimeProvider) so tests run instantly. - Composition tests:
Polly.TestingexposesGetPipelineDescriptor()to assert which strategies a pipeline contains and how they are configured. - Chaos engineering: Polly v8 includes chaos strategies (fault, outcome, latency, behaviour) to inject failures into real environments on purpose.
- Load tests during failure: fail a dependency while under load and check the retry rate doesn't overload it when it recovers.
14. Anti-patterns to avoid
Retrying everything
The default ShouldHandle handles nearly any exception, including bugs and validation errors. Narrow it to transient faults.
Retrying forever
Unlimited retries hide outages and pin threads and memory. Use a cap and a total timeout; let a circuit breaker take over.
No delay, no jitter
Immediate, synchronised retries create retry storms. Use exponential backoff with jitter.
Retry at every layer
Nested retries multiply calls (4 × 4 × 4 = 64). Retry at one layer; let inner layers fail fast.
Retrying non-idempotent calls blindly
Charging a card twice is worse than an error message. Use idempotency keys or don't retry.
Ignoring Retry-After
A throttling service tells you when to come back. Retrying sooner just earns more 429s.
Retry as a scheduler
Using a 24-hour retry delay to run something daily holds memory and doesn't survive restarts. Use Quartz.NET, Hangfire or a hosted service.
Retry instead of scaling
If a service is always busy, retries just add load. Scale or partition it.
Swallowing cancellation
Catching OperationCanceledException and retrying ignores the caller who gave up. Always pass the CancellationToken through.
Side work in OnRetry
OnRetry doesn't run before the first attempt. Put “before each attempt” work inside the executed callback, and retry conditions in ShouldHandle, not OnRetry.
Hand-rolled retry: what goes wrong
15. When to use it — and when not to
USERetry is a good fit when
- Your app talks to a remote service or resource that can have transient faults.
- Faults are expected to be short-lived, so a repeat could succeed.
- The operation is idempotent, or you can make it so.
- The caller can tolerate a little extra latency.
- The dependency signals throttling (429/503 with
Retry-After).
AVOIDRetry is not useful when
- A fault is likely to be long-lasting — you waste time and resources and hurt responsiveness. Use a circuit breaker.
- Failures aren't transient, such as errors in business logic.
- You are using it as an alternative to fixing scalability problems.
- The operation isn't idempotent and you can't add a key.
- A library below you already retries.
Tune for the caller
| Caller | Retries | Backoff | Total budget |
|---|---|---|---|
| Interactive web/API request | 1–3 | Exponential + jitter, 100–500 ms base | Seconds, then a friendly error or fallback |
| Service-to-service (internal) | 2–3 | Exponential + jitter | Inside the caller's own timeout |
| Background job / batch | 5–10+ | Exponential + jitter with MaxDelay | Minutes; then dead-letter and alert |
| Message consumer | Few in-process | Short | Let the broker redeliver; dead-letter after max delivery count |
16. Real-world scenarios
Twenty situations you'll meet in production, with a verdict and how to configure it. Filter by verdict:
18. Production checklist
Tick these off for each dependency you call.
0 of 12 done
19. Glossary
- Transient fault
- A temporary failure that is likely to succeed if the operation is repeated after a short delay.
- Backoff
- The waiting time between attempts; constant, linear or exponential.
- Jitter
- Randomness added to the backoff so many clients don't retry at the same moment.
- Retry storm / thundering herd
- Synchronised retries from many clients overwhelming a recovering service.
- Retry amplification
- Retries at several layers multiplying the number of calls to the bottom layer.
- Retry budget
- A limit on how many retries a client may make, often a token bucket refilled by successes.
- Idempotent
- Repeating the operation has the same effect as doing it once.
- Idempotency key
- A unique ID sent with each attempt so the server can detect and deduplicate repeats.
- Retry-After
- An HTTP header (seconds or a date) telling the client when to try again.
- Resilience pipeline
- Polly v8's chain of strategies (retry, timeout, circuit breaker…) that wraps your call.
- ShouldHandle
- Polly's predicate that decides which exceptions and results are retried.
- Attempt timeout vs. total timeout
- The limit for one try vs. the limit for the whole operation including retries.
- Circuit breaker
- Stops calls to a failing dependency for a while, then lets a trial request through.
- Hedging
- Sending extra parallel attempts when the first is slow and using the fastest answer.
- Dead-letter queue
- Where messages go after too many failed deliveries, for inspection instead of endless retry.
- Execution strategy
- EF Core's built-in retry mechanism for transient database errors (
EnableRetryOnFailure).
Based on the Azure Architecture Center Retry pattern, the Polly v8 documentation and Microsoft Learn's “Build resilient HTTP apps” guidance, checked in October 2026.