It's not a question of if. It's 2 PM on a Thursday and your payment gateway just returned 503.
You've built your payment system on top of an aggregator or a banking API provider. Clean REST endpoints, decent documentation, a sandbox that mostly works. Payments flow. Life is good.
Then the provider goes down. Not for five minutes — for two hours. During peak processing. On a day when your merchants are expecting €8M in payouts.
What happens next depends entirely on decisions you made six months ago, when everything was working.
Your provider promises 99.9% uptime. That's 8 hours and 46 minutes of downtime per year. Sounds acceptable — until all 8 hours happen on the same day, during your busiest period, and your merchants are calling your operations team asking where their money is.
SLAs tell you what happens to the provider's invoice when they fail. They don't tell you what happens to your business. Your merchants don't know or care that you use a third-party provider. They know you promised to pay them, and you haven't.
Design as if the SLA doesn't exist. Design as if the provider will go down at the worst possible time, because that's exactly when you'll notice.
When a payment API goes down, the failure doesn't stop at the API call. It cascades:
That last one is the killer. The moment your team starts working around your system instead of through it, you've lost control of the data. Everything done manually during the outage needs to be reconstructed and reconciled later — and it always takes longer than the outage itself.
The first technical defence is a circuit breaker. The pattern is simple:
Without a circuit breaker, your system will keep hammering a dead endpoint, consuming connections, filling timeout queues, and potentially bringing down your infrastructure. A cascade failure that starts at the provider and ends at your database connection pool is not theoretical — we've seen it.
The circuit breaker gives you two things: it protects your system from the provider's failure, and it gives you a clean signal that you're in a degraded state.
When the circuit opens, payments don't disappear. They go into a durable queue.
This queue is critical. It represents the business's intent — these payments need to go out, even if they can't go out right now. The queue must be:
When the circuit closes and the provider recovers, the queue drains automatically. But the drain needs to be throttled — dumping 500 queued payments at once when the provider just recovered is a great way to knock it down again.
If your business can't tolerate an outage for the duration of the provider's downtime, you need a secondary path.
This is where architecture gets expensive but honest. A secondary payment channel — a direct bank integration, a second aggregator, a manual payment process with system-level controls — means you can continue operating when your primary provider fails.
The considerations:
Cost vs. resilience: a secondary provider adds integration cost, ongoing maintenance, and usually higher per-transaction fees (because you're sending lower volume through it). Is the cost of the secondary worth the cost of the outage? For a business doing €8M in daily payouts, a two-hour outage has a real cost. Do the maths.
Consistency: if you route some payments through Provider A and some through Provider B, your reconciliation needs to handle both. Bank statements won't tell you which provider was used — they'll just show transactions. Your system needs to track the routing decision and match accordingly.
Automatic vs. manual failover: automatic failover sounds ideal but is genuinely hard. You need confidence that the primary is actually down (not just slow), that the secondary is actually up, and that routing to the secondary won't create duplicate payments (if the primary processed some payments before failing). Manual failover — where an operator makes the decision to switch — is often safer.
The nastiest scenario is not when the provider goes down cleanly. It's when it goes down mid-transaction. You sent a payment instruction. You got a timeout. Did the payment go through or not?
You don't know. And you can't ask, because the provider is down.
This is where idempotency keys earn their keep. Every payment instruction you send should include a unique idempotency key. When the provider recovers and you retry, the idempotency key ensures the payment is processed once, not twice.
But idempotency only works if the provider supports it properly. Check. Don't assume. If your provider doesn't support idempotency keys, you need a different strategy: check the payment status before retrying, or wait for the bank statement to confirm whether it settled.
Sending a payment twice because you couldn't confirm the first one went through is worse than sending it late. Your merchant will be annoyed about a delayed payment. They'll be very annoyed about a duplicate one — and so will your bank.
When payments stop flowing, someone needs to be told. Your system should have an alerting chain:
The worst thing you can do during a provider outage is nothing. Silence breeds panic, and panic breeds manual workarounds that create bigger problems tomorrow.
The outage ends. The provider is back. The queue drains. Payments start flowing again. Everyone exhales.
But you're not done. The outage created data gaps:
Financial systems should be designed around the assumption that any external dependency will fail. Not might fail — will fail. The question is whether your system degrades gracefully or collapses.
Graceful degradation means:
This isn't pessimism. It's professionalism. Your merchants don't care about your provider's infrastructure problems. They care about getting paid. Design accordingly.
Zenlime builds payment systems that handle provider failures as a normal operating condition. Start a conversation.