The Outbox Pattern for Reliable Events

Every distributed system I have built in fintech eventually runs into the same uncomfortable question: how do you change your database and tell the rest of the world about it at the same time, without ever lying to one or the other? You debit a ledger account and publish a "payment captured" event. You insert a payout record and emit "payout initiated" to a downstream reconciliation service. The work feels atomic in your head, but it is not atomic in your infrastructure. Two systems are involved, and somewhere between the commit and the publish, a process can crash, a broker can be unreachable, or a network partition can swallow your message.

The outbox pattern is the most boring, most reliable answer I know to that problem. It is not glamorous, it does not appear in conference keynotes, and it has saved me from more 2 a.m. incidents than any clever piece of distributed consensus machinery. This is a practical walkthrough of why it works, how I implement it, and the operational realities that the textbook diagrams leave out.

The dual-write problem

The trouble starts the moment a single business operation needs to update two stores of truth. In our world that is almost always a relational database and a message broker. You want to record that money moved and you want to notify the systems that care. The naive implementation does both in sequence inside the same method: write the row, then call the broker. It looks correct in code review and works flawlessly in the demo.

It fails in production because there is no transaction spanning the database and the broker. If the database commit succeeds but the publish fails, you have state with no event: the reconciliation service never learns about the payout. If the publish succeeds but the commit rolls back, you have an event with no state: downstream systems act on a payment that, as far as your own ledger is concerned, never happened. In a regulated environment the second failure is the one that ends careers, because you have told another party something false and acted on it.

People reach for distributed transactions to fix this, and I understand the instinct. But two-phase commit across a database and a modern message broker is either unavailable, badly supported, or operationally brittle. It couples the availability of two systems together and locks resources across a network round trip. I have never seen it survive contact with real traffic in a payments path. The outbox pattern sidesteps the whole question by refusing to do two writes in the first place.

One transaction, one source of truth

The core idea is almost insultingly simple. Instead of writing to the database and then publishing to the broker, you write to the database twice in the same transaction: once to your business table, and once to an outbox table that holds the event you intend to publish. Because both writes are in one local transaction against one database, they are genuinely atomic. Either the payout row and its event both exist, or neither does.

A separate process reads unpublished rows from the outbox table and pushes them to the broker, marking each as sent once the broker acknowledges. The clever part is the inversion: you have replaced an unsolvable cross-system atomicity problem with a perfectly ordinary single-database transaction plus an asynchronous relay. The database becomes the single source of truth for both your state and your intent to communicate.

The outbox does not make publishing reliable. It makes the decision to publish durable, and durability is something a database already guarantees. Everything else is just retry.

Schema and the transactional write

I keep the outbox table deliberately generic so any aggregate can use it. The columns I always include are an identifier, the event type, a serialized payload, a creation timestamp, a processed timestamp that stays null until relayed, and an attempt counter. I add an aggregate identifier too, because ordering per aggregate matters far more than global ordering, and I will come back to that.

The write itself happens inside the same transaction as the business change. In .NET with Entity Framework Core this falls out naturally because EF batches everything in a single SaveChanges into one transaction. The discipline is simply to never publish inline; you only ever insert the outbox row.

public async Task CapturePaymentAsync(Guid paymentId, decimal amount)
{
    await using var tx = await _db.Database.BeginTransactionAsync();

    var payment = await _db.Payments.FindAsync(paymentId);
    payment.Capture(amount); // mutates ledger state

    var evt = new OutboxMessage
    {
        Id = Guid.NewGuid(),
        AggregateId = paymentId,
        Type = "payment.captured",
        Payload = JsonSerializer.Serialize(new
        {
            paymentId,
            amount,
            capturedAt = DateTime.UtcNow
        }),
        CreatedAt = DateTime.UtcNow
    };
    _db.Outbox.Add(evt);

    // One commit. State and intent succeed or fail together.
    await _db.SaveChangesAsync();
    await tx.CommitAsync();
}

Notice what is absent: there is no broker call anywhere in this method. The handler does not know or care whether the broker is healthy. That is the point. Your business logic stays synchronous, fast, and decoupled from the availability of every downstream consumer.

The relay and delivery guarantees

The relay is a background worker that polls the outbox for rows where the processed timestamp is null, ordered by creation time, and publishes them. After the broker acknowledges, it stamps the processed timestamp. If the broker is down, the rows simply stay unprocessed and the next poll picks them up. This is the mechanism that converts a durable record into an actually-delivered message.

The guarantee you get is at-least-once delivery, and you must internalize that phrase. The relay can publish a message, crash before it records the processed timestamp, and publish the same message again on restart. This is not a flaw to be engineered away; it is the honest contract of any system that survives crashes. The query that drives the relay is the kind of thing I want every engineer on the team to be able to read:

SELECT id, aggregate_id, type, payload
FROM outbox
WHERE processed_at IS NULL
  AND attempts < 10
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;

The FOR UPDATE SKIP LOCKED clause is what lets me run several relay instances concurrently without them fighting over the same rows. Each worker grabs a batch, locks it, and the others skip straight past the locked rows to find their own work. It is a small phrase that turns a single-threaded bottleneck into a horizontally scalable one.

Why consumers must be idempotent

Because delivery is at-least-once, every consumer on the other end has to be idempotent. Processing the same "payment.captured" event twice must produce the same result as processing it once. This is non-negotiable, and it is the part teams most often underestimate when they adopt the pattern. The outbox guarantees the message arrives; it does nothing to guarantee it arrives exactly once.

In practice I make consumers idempotent in a few well-worn ways:

• Carry a stable event identifier and have the consumer record processed identifiers, rejecting duplicates it has already seen.
• Make the downstream operation naturally idempotent, for example an upsert keyed on the payment identifier rather than a blind insert.
• Use conditional updates that only apply when the current state matches what the event expects, so a replayed event is a harmless no-op.
• Treat side effects to external parties, like sending an email or calling a partner API, as the riskiest case and gate them behind their own deduplication store.

I have learned to write the idempotency requirement directly into the consumer's acceptance criteria. If a developer cannot tell me what happens when their handler sees the same event twice, the handler is not finished, no matter how green the tests are.

Ordering and partitioning

Strict global ordering is expensive and almost never what the business actually needs. What it needs is ordering per entity: for a single payment, "authorized" must arrive before "captured" must arrive before "refunded". This is why I store an aggregate identifier on every outbox row. The relay can publish to a broker partition keyed on that aggregate, which preserves per-entity order while allowing unrelated payments to flow in parallel.

The polling relay, ordered by creation time, gives you a reasonable approximation of order on the producing side. But ordering can still be disturbed by retries: a failed message that is retried later can land after messages that were created after it. The clean answer is to refuse to advance an aggregate's events past a stuck one, but that trades throughput for strictness. For most flows I let consumers tolerate mild reordering through their idempotency logic rather than enforcing rigid sequencing everywhere, and reserve strict per-aggregate gating for the handful of flows where it genuinely matters.

Polling versus change data capture

There are two ways to get rows out of the outbox table. The first is the polling relay I have described: a worker that queries on an interval. It is simple, easy to reason about, easy to operate, and it has no dependency beyond your database. The trade-off is that polling adds latency and load; a tight poll loop is wasteful, and a slow one delays events.

The second approach is change data capture, where a tool tails the database transaction log and emits a message for every committed outbox insert. This removes the polling latency and the query load entirely, and it scales beautifully. The cost is operational complexity: you now run and monitor a log-tailing pipeline, manage its offsets, and understand its failure modes. My rule of thumb is to start with polling because it gets you the correctness guarantee immediately, and graduate to change data capture only when measured latency or load actually justifies the extra moving part. Do not buy the complexity before you need it.

Operational realities the diagrams omit

The pattern is simple; running it well is where the real work lives. The first thing that will bite you is table growth. A busy outbox accumulates millions of processed rows, and your relay query slows to a crawl because the index has to scan past a graveyard of already-sent messages. I always pair the outbox with a partial index on unprocessed rows and a scheduled job that archives or deletes rows once they are safely processed and past any audit retention window.

The second is poison messages. A row that always fails to publish, perhaps because its payload is malformed or a consumer rejects it, will be retried forever and can block a per-aggregate stream. This is what the attempt counter is for: past a threshold I move the row to a dead-letter state and raise an alert rather than spinning indefinitely. The third is observability. I want metrics on outbox depth, oldest unprocessed age, and relay throughput, because a silently stalled relay looks exactly like a healthy system right up until the backlog becomes an incident. An alert on "oldest unprocessed event older than thirty seconds" has caught broker problems for me long before any customer noticed.

Conclusion

The outbox pattern earns its place in my toolkit precisely because it is unexciting. It replaces a genuinely hard distributed-systems problem, atomic dual writes, with two things databases already do superbly: local transactions and durable storage. The price you pay is accepting at-least-once delivery and the discipline of idempotent consumers, and that is a price worth paying because it forces a healthier design on the whole system. If you build payments infrastructure and you are still publishing events inline after a commit, the outbox is the cheapest reliability upgrade you can make, and the one I would reach for first.