The Saga Pattern for Distributed Transactions

A few years ago I watched a payout run leave forty-two merchants with money debited from our settlement account but never credited to their bank. The culprit was not a bug in any single service. Each service worked exactly as designed. The problem was that we had five services participating in one logical transaction, no shared database, and a naive assumption that if step three succeeded, steps four and five would too. Step four timed out. Nobody undid step three. That is the whole story of distributed transactions in one sentence.

The Saga pattern is how you stop that from happening. It is not glamorous and it does not give you the tidy guarantees of a single database transaction. What it gives you is a disciplined way to fail, which in payments is worth more than almost anything else.

Why two-phase commit quietly lost

The textbook answer to a transaction spanning multiple resources is two-phase commit. A coordinator asks every participant to prepare, everyone votes, and if all vote yes the coordinator tells them to commit. On paper it preserves atomicity across services. In practice I have not deployed it in anger in over a decade, and I would push back hard if a team proposed it for a new payments flow.

The reason is blocking. During the window between prepare and commit, every participant is holding locks and waiting on a single coordinator. If that coordinator dies at the wrong moment, participants sit there holding resources, unsure whether to commit or roll back. In a system where a ledger row being locked means a merchant cannot see their balance, that is not an edge case you can wave away. Add the reality that most modern participants are HTTP services and message brokers that were never built to enlist in an XA transaction, and 2PC stops being an option long before you finish drawing the diagram.

What a saga actually is

A saga breaks one distributed transaction into a sequence of local transactions, each in its own service and its own database. Every step that changes state has a paired compensating action that semantically undoes it. You do not roll back in the database sense. You move forward through a sequence of corrections. If reserving inventory succeeded but charging the card failed, you do not un-write the reservation with some magic rollback; you run a compensating step that releases the reservation.

The crucial mental shift is that a saga trades atomicity for eventual consistency. There will be moments, sometimes seconds, sometimes longer, where the system is in a partially completed state. A payment is authorized but not captured. A refund is initiated but not yet reflected in the merchant balance. You have to design as if those windows are visible, because they are.

A saga does not prevent inconsistency. It bounds it, makes it observable, and guarantees you have a defined path back to a consistent state. That is a very different promise than ACID, and pretending otherwise is how teams get burned.

Orchestration versus choreography

There are two ways to wire the steps together, and I have strong opinions about which one to reach for first. Choreography means each service listens for events and reacts, publishing its own events in turn. There is no central brain. Orchestration means a dedicated coordinator holds the sequence, calls each service, and decides what happens next based on the result.

Choreography looks elegant in a diagram with three services. It becomes a nightmare at eight, because the business process no longer lives anywhere. It is smeared across event subscriptions in a dozen codebases, and the only way to answer "what happens after a payment fails" is to trace event handlers across repositories. I once inherited a refund flow built entirely with choreography, and it took two engineers a full day to reconstruct the actual sequence from the code. Nobody could draw it from memory because nobody owned it.

For anything with real business logic, I default to orchestration. The orchestrator is an honest place where the process lives. You can read it, test it, and put a state machine diagram next to it that actually matches reality. The cost is one more service to run and a coordinator you must make durable. That cost is almost always worth paying.

Compensation is the hard part

Everyone gets the happy path right. The steps that matter are the compensations, and they are harder than they look for one blunt reason: some things cannot be cleanly undone. You can release a reservation. You cannot un-send an email that told a merchant their payout is on the way. You cannot always reverse a captured card payment without fees, and in some schemes you cannot reverse it at all within the same window.

This forces you to think about the order of your steps. Put the reversible, cheap-to-compensate steps early. Put the irreversible or externally visible steps as late as possible, ideally last, so that by the time you take them the risk of needing to compensate is near zero. In our payout redesign we moved the actual bank transfer to the final step precisely because it is the one action we genuinely cannot pull back once the file leaves us.

• Design every compensation to be idempotent. It will be retried, and a compensation that double-refunds is worse than the original failure.
• Make compensations tolerant of a step that never fully committed. A release-reservation that throws because there is no reservation is a landmine.
• Accept that some steps are non-compensatable, and structure the saga so those run last.
• Log the compensation as a first-class business event, not a silent cleanup. Finance will ask about it later.

Idempotency and the double-charge

Networks lie. A step returns a timeout, but the work actually completed on the other side. If your orchestrator retries blindly, you now have a double charge, and a very unhappy compliance conversation. Every state-changing step in a saga needs an idempotency key so the receiving service can recognize a repeat and return the original result instead of doing the work twice.

In practice this means the orchestrator generates a stable key per step per saga instance and the participant persists that key alongside the outcome. This is not optional decoration. A saga without idempotency at every step is a machine for producing duplicate charges the first time a broker redelivers a message. And brokers redeliver constantly.

public async Task<StepResult> ChargeCard(SagaContext ctx)
{
    // Idempotency key is stable per saga step, so a retry is a no-op.
    var key = $"charge:{ctx.SagaId}:{ctx.StepId}";

    var existing = await _payments.FindByIdempotencyKey(key);
    if (existing is not null)
        return StepResult.From(existing); // already done, return prior outcome

    try
    {
        var charge = await _gateway.Charge(ctx.Amount, ctx.CardToken, key);
        await _payments.Persist(key, charge);
        return StepResult.Success(charge.Id);
    }
    catch (GatewayDeclinedException ex)
    {
        // Business failure -> trigger compensation of earlier steps.
        return StepResult.Compensate(ex.Reason);
    }
}

The state you must persist

An orchestrator that keeps saga state in memory is a demo, not a system. The whole point is surviving process crashes, deployments, and the machine simply vanishing mid-flow. Which means the current step, the outcome of every completed step, and enough context to run any compensation must live in durable storage that you write to before and after each step.

We store each saga instance as a row plus an append-only log of step transitions. On startup the orchestrator scans for sagas that are in flight and resumes them. This recovery path is the part teams skip because it is boring and rarely exercised, and it is also the part that saves you at three in the morning. Test it deliberately. Kill the orchestrator mid-saga in a staging environment and confirm it picks up exactly where it left off, without re-running a completed capture.

Timeouts and stuck sagas

A saga can get stuck. A downstream service is down, a step neither succeeds nor cleanly fails, and the instance sits half-finished. If you do not plan for this, stuck sagas accumulate silently until someone notices a merchant balance is wrong days later. Every step needs a timeout, and every timeout needs a decision: retry with backoff, compensate, or escalate to a human.

My rule is that no saga is allowed to be stuck indefinitely without raising an alert. We set a maximum lifetime per saga type. A payout saga that has not reached a terminal state within its budget pages the on-call engineer with the saga id and its current step. Sometimes the right answer is automated compensation. Sometimes a payment genuinely left the building and the only correct action is a human deciding what to do. Encoding "give up and ask a person" as a legitimate terminal state is not an admission of defeat; it is the responsible design for money movement.

When not to reach for one

Sagas are a real cost. More code, more state, more failure modes to reason about, and an eventual-consistency window your product people have to understand and accept. If your transaction fits inside a single service and a single database, use a database transaction and go home early. Do not distribute a transaction just because microservices are fashionable.

I have also killed saga proposals where the honest answer was that the two services should have been one. If two components are so tightly coupled that they must always change state together, that is a data-modeling signal, not a coordination problem. Reach for a saga when you genuinely have independent services with their own data that must cooperate on an outcome, and the business can tolerate a brief inconsistency window. If either of those is untrue, you are adding complexity to solve a problem you created.

Conclusion

The thing I wish someone had told me before that forty-two-merchant incident is that a saga is not a technique for making distributed transactions feel safe. It is a technique for making failure explicit and recoverable. The value is not in the happy path, which any framework will hand you. It is in the compensations, the idempotency keys, the persisted state, and the alert that fires when a saga is stuck. Get those right and you will sleep. Skip them and you will eventually be reconciling ledgers by hand at midnight, wondering where the money went. I have done both, and I strongly recommend the first one.