Scheduler Agent Supervisor

1. Why this pattern exists

A single business task often needs several remote calls: reserve stock, charge a card, book a courier. Each call can fail for different reasons, and they fail independently. You cannot wrap three different services in one ACID database transaction, so you need another way to make the whole task end up either fully done or cleanly undone.

The pattern was first described by Clemens Vasters (Microsoft) from his work on early Windows Azure, where queues and storage could misbehave and machines could be recycled at any time. It is documented today in the Azure Architecture Center's cloud design patterns, and the same idea sits inside modern workflow engines such as Azure Durable Functions.

2. Mental model: a restaurant kitchen

If you remember one picture, remember this one. A dinner order is a multi-step task handled by different stations.

Notice what the expediter does not do: they don't cook, and they don't decide the recipe. They only notice lateness and trigger "try again" or "give up". If a cook faints, the restaurant (the platform) brings in another cook; the expediter just makes sure the late ticket gets redone.

3. The actors and their exact responsibilities

These are logical roles. Physically they can be separate services, one process, functions, or containers; several logical agents can live in one web service.

4. The state store, field by field

Microsoft's reference example stores one record per order. In a multi-step workflow you store one record per step. These fields carry the whole pattern:

The life of one step

5. Failure simulator: break things and watch recovery

An order runs through three steps, each owned by an Agent that wraps one remote service. Press Play for the happy path, or arm a fault first: it hits the next step that gets dispatched. Every beat highlights the matching lines in the C# code on the right. Click any line of the event log to jump back to that explanation.

6. Failure catalogue: what breaks and who fixes it

7. Key concepts you need (glossary)

8. Tuning the timings (interactive)

Three numbers decide how fast you recover and how much load you add. Move the sliders to see the trade-offs.

9. Building it in C# / .NET

The simulator's code panel holds two complete, annotated listings. Here is how the pattern's roles map onto each, plus the pieces worth copying.

Atomic claim: only one Scheduler wins

// Azure Table Storage or Cosmos DB: optimistic concurrency with the ETag
public async Task<bool> TryClaimAsync(StepEntity row, string me, DateTimeOffset deadline, CancellationToken ct)
{
    row.LockedBy = me;
    row.Status = "Processing";
    row.CompleteBy = deadline;
    try
    {
        await _table.UpdateEntityAsync(row, row.ETag, TableUpdateMode.Replace, ct);
        return true;                       // we own the step
    }
    catch (RequestFailedException e) when (e.Status == 412)
    {
        return false;                      // another instance won the race
    }
}

// SQL Server equivalent: claim only if still free; @@ROWCOUNT = 1 means you own it
// UPDATE StepState SET LockedBy = @me, Status = 'Processing',
//        CompleteBy = DATEADD(second, 30, SYSUTCDATETIME())
// WHERE OrderId = @id AND StepNo = @n AND Status = 'Pending' AND LockedBy IS NULL;

Durable Functions: things that changed recently

• Use the isolated worker model. Microsoft's migration guide states the in-process model reaches end of support on 10 November 2026. Packages: Microsoft.Azure.Functions.Worker.Extensions.DurableTask; types come from Microsoft.DurableTask (TaskOrchestrationContext, TaskOptions, RetryPolicy).
• Failures arrive as TaskFailedException in the isolated model (in-process used FunctionFailedException).
• Durable Task Scheduler is now the recommended, fully managed backend. Its Dedicated SKU went GA in November 2025 and its pay-per-use Consumption SKU went GA in spring 2026. It also backs the Durable Task SDKs on Container Apps, AKS and App Service.
• Netherite backend support ends 31 March 2028; Microsoft suggests evaluating Durable Task Scheduler instead.
• Orchestrators must be deterministic: use ctx.CurrentUtcDateTime not DateTime.UtcNow, ctx.NewGuid() not Guid.NewGuid(), no direct I/O, and log through ctx.CreateReplaySafeLogger.
• Don't retry business errors. Separate transient failures (retry) from validation or authorization failures (fail fast, compensate). Compensation activities must be idempotent because they can run more than once.

Resilience inside an Agent (Polly v8)

Microsoft.Extensions.Http.Polly is deprecated. Use Microsoft.Extensions.Resilience (general pipelines) and Microsoft.Extensions.Http.Resilience (HttpClient). Order matters: a timeout outside the retry is a total budget; a timeout inside it is per attempt.

builder.Services.AddHttpClient<CardGatewayClient>()
    .AddStandardResilienceHandler();      // rate limiter, total timeout, retry, circuit breaker, attempt timeout

builder.Services.AddResiliencePipeline("warehouse", p => p
    .AddTimeout(TimeSpan.FromSeconds(25))                // total budget (keep it below CompleteBy)
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 4,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        ShouldHandle = new PredicateBuilder().Handle<HttpRequestException>()
    })
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5, MinimumThroughput = 10,
        SamplingDuration = TimeSpan.FromSeconds(30),
        BreakDuration = TimeSpan.FromSeconds(30)
    })
    .AddTimeout(TimeSpan.FromSeconds(5)));               // per attempt

Which flavour should I choose?

10. When to use it, and when not to

11. Real-world scenarios

The same roles appear across very different industries. For each, look at the steps, the deadline, and how undo works.

12. Related patterns and how they differ

Well-Architected view: the pattern supports Reliability (detect failures, reroute to a healthy agent, self-heal) and Performance Efficiency (route to agents with capacity, prioritise critical work). Weigh it against added complexity and operating cost.

13. Pitfalls and how to test

[Fact]
public async Task Expired_step_is_reset_for_retry()
{
    var time  = new FakeTimeProvider(DateTimeOffset.Parse("2026-10-25T10:00:00Z"));
    var store = new InMemoryStateStore();
    await store.AddAsync(StepState.Processing("ORD-1", stepNo: 2,
        completeBy: time.GetUtcNow().AddSeconds(30)));

    var supervisor = new Supervisor(store, new AlwaysLeader(), new NullAlerts(), time);
    await supervisor.StartAsync(CancellationToken.None);

    time.Advance(TimeSpan.FromSeconds(46));          // past CompleteBy + one 15 s tick
    var row = await store.WaitForChangeAsync("ORD-1", 2);

    Assert.Equal(StepStatus.Pending, row.Status);
    Assert.Equal(1, row.FailureCount);
    Assert.Null(row.LockedBy);
}

// Chaos in staging: 10% very slow calls, 5% faults
builder.Services.AddResiliencePipeline("payment-chaos", p => p
    .AddChaosLatency(0.10, TimeSpan.FromSeconds(40))
    .AddChaosFault(0.05, () => new HttpRequestException("chaos")));

14. Cheat sheet

Golden rules: state lives in the store, never in memory · every step is idempotent · every completed step has a recorded way to undo it · deadlines are generous but finite · something always tells a human when automation gives up.