Cross-border payments are a sequence of steps that happen at different speeds, controlled by different parties, any of which can fail independently and at the worst possible moment. A simple request-response model fights that reality at every turn. An event-driven model embraces it. After building payment systems both ways, in anger and at scale, I will not go back, and this primer is the explanation I give engineers joining the team.
Why payments are naturally event-driven
A single cross-border transfer might involve initiation, compliance screening, foreign-exchange conversion, settlement with a banking partner in another country, and final confirmation back to the customer. These steps do not happen instantly, and they certainly do not happen in a tidy synchronous line. Compliance screening might take milliseconds or might take a human review measured in hours. A partner bank settles on its own schedule, not yours.
Modeling each step as an event, PaymentInitiated, ComplianceCleared, FxLocked, SettlementConfirmed, matches how the money actually moves through the real world. The architecture stops fighting the domain and starts mirroring it, and an enormous amount of accidental complexity simply evaporates.
The state machine underneath
Events without a state machine are just chaos with a message broker attached. Every payment is a state machine, and the events are the transitions between states. The discipline that makes this safe is to make illegal transitions impossible to represent. A payment cannot move to settled before compliance has cleared, and your code should refuse to even construct that state, not merely check for it later and hope.
Model the states explicitly and exhaustively. The bugs that genuinely scare me are not the ones that crash; they are the ones where money quietly ends up in a state the designers never imagined could exist.
I represent the state machine in code as an explicit enum of states and a single function that decides which transitions are legal from each one. Everything else routes through that function, so there is exactly one place in the system that knows the rules, and there is no path around it.
Idempotency and ordering
In any real message broker, messages get redelivered and can arrive out of order. This is not an edge case to handle someday; it is the normal operating condition, and a design that assumes exactly-once, in-order delivery is a design that will corrupt money in production. Two rules keep this survivable.
Enjoying this article?
Get more like it in your inbox — practical engineering leadership, fintech, and AI. No spam, unsubscribe anytime.
- Every consumer must be idempotent: processing the same event twice produces exactly the same result as processing it once.
- Never assume ordering. Carry enough state in the event or check enough state in the store to reject a transition that does not make sense yet.
If a SettlementConfirmed event somehow arrives before ComplianceCleared due to a redelivery quirk, the state machine rejects it because settling an uncleared payment is an illegal transition. The system stays correct even when the message bus misbehaves, which it will.
The audit trail you get for free
Here is the benefit that is underappreciated outside of regulated industries. An event log is, by its nature, an audit log. When a regulator or a customer or your own finance team asks what happened to a specific payment, you can replay the exact sequence of events with timestamps and reconstruct the truth precisely.
In a regulated business this built-in traceability is worth as much as the architectural flexibility, sometimes more. I have sat in audits where the difference between a short conversation and a painful one was entirely down to being able to say "here is the complete, ordered, timestamped history of this transaction" and mean it.
Where it costs you
I would be lying if I sold this as free. Event-driven systems are harder to reason about locally. You trade the comforting simplicity of a single call stack you can step through in a debugger for the resilience of decoupled steps you have to trace across services. Debugging spans multiple components. Eventual consistency genuinely confuses newcomers, who expect a write to be immediately visible everywhere. And you need real, sustained investment in observability or you will be blind.
For a simple CRUD application this trade is not worth it, and reaching for events there is resume-driven development. For money that crosses borders, partners, and time zones, it is the only model that has held up for me under real load and real failure.
The observability requirement
Because you have given up the call stack, you must buy back visibility deliberately. Every event carries a correlation ID that ties the whole payment together across services. Every consumer logs the transition it performed with that ID. A single dashboard can then show the complete life of any payment as a sequence of events, which is the thing that makes the architecture debuggable rather than merely elegant. Skip this, and event-driven design becomes a way to distribute your bugs across more services than you can hold in your head.

Conclusion: match the architecture to the domain
Cross-border payments are asynchronous, multi-party, and failure-prone by their fundamental nature. An event-driven architecture, anchored by an explicit state machine, made idempotent against redelivery, and backed by serious observability, matches that reality instead of fighting it. It costs you local simplicity and buys you resilience, auditability, and a system that tells the truth about what happened to every shilling that passed through it. For this domain, that is the right trade, every time.
Get new posts in your inbox
Occasional, practical notes on engineering leadership, fintech, and building with AI. No spam, unsubscribe anytime.
Comments (10)
Leave a Comment
Ryan Young
July 22, 2026
Does the "Conclusion: match the architecture to the domain" still hold on a 6-engineer team? We're at the awkward middle and some of these patterns feel like they need a dedicated SRE to run properly.
Amina Sani
July 8, 2026
Quick q on "Conclusion: match the architecture to the domain" — how do you handle duplicate events when the partner API sends duplicate callbacks? We're on AWS org unit split and ops keep asking for manual replay tooling.
David Davis
July 6, 2026
Quick q on "Conclusion: match the architecture to the domain" — how do you handle duplicate events when the partner API sends duplicate callbacks? We're on whatever the seniors set up and ops keep asking for manual replay tooling.
Oluwaseun Bakare
June 29, 2026
Quick question on "The observability requirement" — does the pattern hold when you have to support both sync and async callers? We keep running into the bursty case and the textbook answers do not always survive contact.
Rupert Redgrave
June 18, 2026
Good topic. Independent consultant, 6 due-diligence engagements a year across UK/EU fintechs here. What we do differently: run a shadow processor comparing against production for a week before cut-over on nothing fixed — I evaluate their stack. It is not universally better; operational complexity is real, but the testability is dramatically better and that pays for itself the first time you have to answer a CBN question at 5am.
Poppy Sutherland
June 17, 2026
Quick question on "Conclusion: match the architecture to the domain" — does the pattern hold when the schema is versioned by a partner? We keep running into the partner-driven case and the textbook answers do not always survive contact.
Katie Martinez
June 15, 2026
Enjoyed this one. A small nit on "The state machine underneath": worth mentioning FIFO ordering under failover — otherwise the pattern degrades under real load.
Chris White
June 14, 2026
SRE at a ledger-as-a-service startup. We hit this exact thing with multi-region failover last May — manual toil was the presenting symptom, and the "Where it costs you" section is basically how we untangled it. Ended up adding a materialised view on Istio, cut manual replays by 69%. For context: 10-service graph.
Zainab Lawal
June 14, 2026
Good topic. Fractional CTO — done 4 fintech engagements in the last 5 years here. What we do differently: push the matching into the database instead of pulling into app code on Terraform. It is not universally better; the on-call story is worse, but the audit story is dramatically better and that pays for itself the first time you have to answer a SOC 2 auditor question at 4am.
Arthur Bennett
June 12, 2026
Senior eng at a London fintech scaleup, working mostly on payout rails. We hit this exact thing with GDPR SAR requests last sprint — tail latency was the presenting symptom, and the "Where it costs you" section is basically how we untangled it. Ended up carving off a background worker on Postgres 15, cut error rate by 74%. For context: 2160 rps.
