Building an Idempotent Refund Flow

The first time a refund double-fired in production, it cost us about 4,200 euros before anyone noticed. A merchant's support agent clicked the refund button, the request timed out at our gateway, the agent clicked again, and both requests eventually landed. Two refunds, one order. The customer was delighted. Our reconciliation team was not.

That incident is why I have strong feelings about idempotency in refund flows. Refunds move real money out the door, they are often triggered by humans clicking buttons under stress, and they sit on top of networks that fail in the most inconvenient ways. If you get this wrong, you do not find out in staging. You find out from a finance analyst three days later, holding a spreadsheet.

Why refunds are harder than payments

People assume a refund is just a payment in reverse. It is not, and treating it that way is how teams end up in trouble. A payment has a natural anchor: the customer's order, the checkout session, the authorization. There is usually one obvious thing that should happen exactly once, and the whole industry has spent two decades building rails around that assumption.

Refunds are messier because they are partial, repeatable, and asynchronous all at once. A single order can legitimately be refunded five times: three line items on Monday, shipping on Tuesday, a goodwill credit on Friday. Each of those is a distinct, valid operation against the same order. So you cannot simply say "this order has been refunded" and short-circuit. You have to reason about which specific refund attempt this is, and whether that exact attempt has already happened.

On top of that, the money movement is frequently out of your hands. You call an acquirer or a card scheme, they accept the request, and the actual settlement happens hours later in a batch. Your system has to stay correct across that whole window, including the part where your process crashes halfway through and a retry mechanism wakes up and tries again.

What idempotency actually means here

Idempotency is one of those words that gets nodded at in design reviews and then quietly ignored in the code. The working definition I hold people to is blunt: if the same logical refund request arrives more than once, the observable effect on money and on state must be identical to it arriving exactly once. Not "roughly the same". Identical.

The key phrase is "same logical request". That is the thing you have to define, and it is where most implementations go wrong. It is not the HTTP request. It is not the row in your queue. It is the caller's intent: this agent, refunding this amount, against this order, for this reason. Two byte-identical HTTP payloads can be two different intents (a genuine second partial refund), and two very different-looking payloads can be the same intent (the same click, retried through a different code path). You need the caller to tell you which is which, and the mechanism for that is the idempotency key.

The idempotency key is a promise from the caller: "if you see this key again, it is me repeating myself, not me asking for something new." Your job is to hold them to that promise even when their client library retries behind their back.

The idempotency key belongs to the caller

I insist that the client generates the idempotency key, not the server. This feels backwards to people the first time, because the server is the source of truth for everything else. But the whole point of the key is to survive a network failure where the caller never learned the outcome. If the server mints the key and the response is lost, the caller has nothing to send back on retry. The key has to be born on the client, before the first attempt, and reused on every retry of that same intent.

A good key is a UUID the client generates and stores alongside the pending refund. A bad key is something like the order ID, or a hash of the amount and order, because those collide across legitimately-different refunds. I have seen a team use "orderId + amount" as the key, and then a customer who was refunded 10 euros twice for two separate reasons got only one refund. The second request looked like a duplicate. It was not. That bug is subtle precisely because it works fine in every demo.

My rule of thumb for what a refund idempotency key must capture:

• The specific refund intent, not the order and not the customer.
• Enough uniqueness that two genuinely different refunds never collide (a client-side UUID does this for free).
• Stability across retries, so the same intent always carries the same key no matter which code path retries it.
• A reasonable lifetime, so you can expire keys after, say, 24 or 48 hours rather than storing them forever.

Store the key before you touch the money

The ordering here is the entire game. The naive implementation checks a cache, sees no key, calls the acquirer, then writes the key. Every gap between those steps is a window where a concurrent duplicate slips through. Two requests both check, both see nothing, both call the acquirer. Congratulations, you have rebuilt the 4,200 euro bug.

The fix is to make the key claim atomic and to do it first, using the database as the arbiter rather than an application-level check. A unique constraint on the idempotency key is the cheapest, most reliable lock you will ever deploy. You insert the key in a "pending" state inside a transaction. If the insert succeeds, you own this refund and you proceed. If it violates the unique constraint, someone else owns it, and you go read their result instead of calling the acquirer.

-- One row per logical refund intent.
CREATE TABLE refund_attempts (
    idempotency_key  UUID        NOT NULL PRIMARY KEY,
    order_id         BIGINT      NOT NULL,
    amount_minor     BIGINT      NOT NULL,   -- store in minor units, never floats
    currency         CHAR(3)     NOT NULL,
    status           VARCHAR(16) NOT NULL,   -- pending | succeeded | failed
    acquirer_ref     VARCHAR(64) NULL,       -- populated once the acquirer responds
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

Notice the amount is stored in minor units as a BIGINT. If you take one thing from this section, take that: money is integers. I have spent too many evenings chasing a one-cent reconciliation break that turned out to be a double rounded somewhere upstream. Floats have no place anywhere near a refund ledger.

The happy path and the crash in the middle

Here is the flow I actually ship. The C# is simplified, but the shape is real and the ordering is deliberate.

public async Task<RefundResult> RefundAsync(RefundCommand cmd)
{
    // 1. Claim the key. This is the atomic gate.
    var claimed = await _repo.TryInsertPendingAsync(cmd.IdempotencyKey, cmd);
    if (!claimed)
    {
        // Someone already owns this intent. Return their outcome.
        var existing = await _repo.GetAsync(cmd.IdempotencyKey);
        if (existing.Status == RefundStatus.Pending)
            throw new RefundInProgressException(); // caller should retry later
        return existing.ToResult();
    }

    // 2. We own it. Call the acquirer, passing OUR key downstream too.
    try
    {
        var resp = await _acquirer.RefundAsync(cmd, cmd.IdempotencyKey);
        await _repo.MarkSucceededAsync(cmd.IdempotencyKey, resp.Reference);
        return RefundResult.Succeeded(resp.Reference);
    }
    catch (AcquirerDeclinedException ex)
    {
        await _repo.MarkFailedAsync(cmd.IdempotencyKey, ex.Reason);
        return RefundResult.Failed(ex.Reason);
    }
    // Note: on a transient/network error we do NOT mark failed.
    // We leave it pending and let a reconciler settle it.
}

The interesting case is the one that looks like a gap: what happens if the process dies right after the acquirer call but before we write the result? The row is stuck in "pending". A retry from the caller comes in, hits the unique constraint, sees "pending", and gets told to wait. That is correct behaviour. What resolves the stuck row is not the retry, it is a separate reconciler that queries the acquirer by our idempotency key and finalizes the state. Which brings me to the part everyone forgets.

Pass your key all the way down

Your idempotency guarantee is only as strong as its weakest hop. If you dedupe perfectly at your API but then call the acquirer without an idempotency key of your own, you have just moved the double-refund risk one layer down where you can no longer see it. Every serious payment provider I have worked with supports an idempotency header on refund endpoints. Use it. Pass a deterministic key derived from your own attempt, so that when your reconciler retries the acquirer call, the acquirer also recognizes the repeat.

This is the difference between an idempotency story that survives an audit and one that merely survives the demo. When an acquirer sees the same refund key twice, it returns the original result instead of moving money again. That is a safety net under your safety net, and it costs you nothing but a header. Skipping it is a decision I have never seen age well.

The reconciler is not optional

Distributed systems do not have a clean "the request finished" moment; they have a request that either finished or is still in a state you have not observed yet. A refund flow that only ever settles state on the synchronous path is a refund flow that will accumulate stuck-pending rows until someone gets paged. You need a background job that finds pending attempts older than some threshold and asks the acquirer what actually happened.

Ours runs every couple of minutes and looks at anything pending for more than five. It queries the acquirer using the same idempotency key we sent originally, reads the real status, and writes it back. In practice this catches a tiny fraction of refunds, well under a percent, but that fraction is exactly the set of cases a human would otherwise have to untangle by hand. I would rather spend one afternoon writing the reconciler than one afternoon a month in reconciliation meetings.

Instrument it well. Emit a metric for pending-age, alert when the oldest pending refund crosses a threshold, and log the idempotency key on every hop so you can trace a single intent end to end. When something does go wrong at three in the morning, the key is the thread you pull to unravel the whole story.

Test the failures, not the happy path

The happy path for refunds is embarrassingly easy to test and tells you almost nothing. The tests worth writing are the ugly concurrency ones. Fire the same idempotency key twice, in parallel, and assert exactly one acquirer call happened and one refund exists. Kill the process between the acquirer call and the state write, restart, replay, and assert no second money movement. Send two genuinely different partial refunds and assert both go through.

My favourite test to write, and the one that has caught the most real bugs, is the interleaved-duplicate test: two threads, same key, artificially delayed so their database reads overlap. If your gate is a SELECT-then-INSERT in application code instead of a unique constraint, this test fails immediately, and it should. I treat a green run of that test as the actual definition of "the refund flow is idempotent". Everything else is commentary.

Conclusion

If I had to compress all of this into one piece of advice, it would be this: make the database refuse the duplicate for you, and stop trying to be clever in application code. Unique constraints do not race, do not forget, and do not deploy a subtle bug on a Friday afternoon. The whole edifice of client-generated keys, pending states, downstream key propagation, and reconcilers exists to lean on that one boring, reliable primitive. Boring is the goal. Nobody has ever written a war story about the refund flow that quietly worked for three years, and that is precisely the outcome you are aiming for.