Worked Design — Idempotent Payment Flow

“Design a payment processing flow.” The whole question is correctness under failure — never double-charge, never lose a payment, stay consistent across your DB and a third-party processor you don’t control. Stack: FastAPI + Postgres + Celery/SQS + Stripe-style processor.

1. Requirements

Functional: a client initiates a payment; the system charges via a payment processor (Stripe/Adyen); the result (success/failure) is recorded and reflected to the client and downstream (order fulfilment, receipts).

Non-functional — these are the design:

Never double-charge — a retry, a network blip, a double-click must not charge twice.
Never lose a payment — if money moved, the system must know.
Consistency across boundaries — your DB, the processor, and downstream services must converge to the same truth even when steps fail mid-flight.
Auditable — every state transition is recorded.

This is a correctness problem, not a scale problem. There’s no clever caching answer — the entire job is handling partial failure.

2. The core threats

Client ──► Your API ──► Payment Processor (Stripe) ──► (money moves)
              │                  │
        could crash here    could time out here
              │                  │  — did the charge succeed? you don't know
        could the client    DB write could fail
        retry?              after the charge succeeded

Every arrow can fail, and a failure between “Stripe charged the card” and “your DB recorded it” is the dangerous one — money moved but your system doesn’t know.

3. Idempotency keys — the foundation

The client generates a unique idempotency key per payment intent (not per HTTP attempt) and sends it with every retry of that payment.

POST /payments
Idempotency-Key: 7f3a9c1e-...        ← same key on every retry of THIS payment
{ "amount": 5000, "currency": "usd", "order_id": "ord_42" }

Server side:

CREATE TABLE payments (
    idempotency_key TEXT PRIMARY KEY,     -- the dedup guarantee
    order_id        TEXT NOT NULL,
    amount          INT  NOT NULL,
    status          TEXT NOT NULL,        -- pending / succeeded / failed
    processor_charge_id TEXT,             -- Stripe's id, once we have it
    response_body   JSONB,                -- the stored response to replay
    created_at      TIMESTAMPTZ DEFAULT now()
);

On request:

INSERT ... ON CONFLICT (idempotency_key) DO NOTHING.
Conflict (key already exists) → this is a retry. If the existing row is succeeded/failed, return the stored response — do not charge again. If it’s still pending, the original is in flight — return “in progress” (409) or wait; do not start a second charge.
No conflict (we inserted) → this is the first time; proceed to charge.

The unique constraint on idempotency_key is the hard guarantee that two concurrent requests for the same payment can’t both proceed.

And pass that same key through to the processor — Stripe et al. accept an Idempotency-Key header and dedup on their side too. Belt and suspenders: even if your dedup has a gap, the processor won’t double-charge.

4. The flow — and the partial-failure handling

1. INSERT payment row (status=pending) keyed by idempotency_key   [atomic claim]
2. Call the processor with the SAME idempotency key
3. Processor responds:
     success → UPDATE row: status=succeeded, processor_charge_id, response_body
     failure → UPDATE row: status=failed, response_body
4. Return the result; emit a "payment.succeeded" event for downstream

Now the failure cases:

Crash between step 1 and step 2 — row is pending, no charge happened. A retry (same key) sees pending; safe to proceed (or a reconciliation job notices a stale pending and either retries or fails it).
Processor call times out (step 2) — you don’t know if the charge succeeded. Do not blindly retry — that risks a double charge. The row stays pending. Resolution: query the processor by the idempotency key (“did a charge with key X succeed?”) — every serious processor supports this lookup. Reconcile the row from the answer.
Crash between step 2 and step 3 — money may have moved but the DB still says pending. The reconciliation job (below) catches this.
Step 3 DB write fails after a successful charge — same: money moved, DB out of sync. Reconciliation catches it. This is why the idempotency key is also sent to the processor and stored — it’s the join key for reconciliation.

The principle: a pending row is a question, not a final state. Something must always resolve it — a retry, or reconciliation.

5. Reconciliation — the safety net

A scheduled job that closes the gap between your DB and the processor’s reality:

Find payments rows stuck in pending past a threshold (e.g. 5 minutes).
For each, query the processor by idempotency key: did a charge with this key succeed?
Update the row to match reality (succeeded with the charge id, or failed).
Additionally: periodically pull the processor’s list of charges and assert every one has a matching row — catches “money moved, we have no record at all.”

Reconciliation is non-negotiable in payments. Idempotency keys prevent double-charges; reconciliation prevents lost payments. You need both.

6. Webhooks — the processor’s async truth

Many processor outcomes arrive asynchronously via webhook (a charge that was pending settles later; a dispute; an async payment method completing). The webhook handler must itself be idempotent:

Webhooks are delivered at-least-once — the processor retries until you 200. So the same charge.succeeded event can arrive twice.
Dedup on the processor’s event id (INSERT ... ON CONFLICT on event_id).
Verify the webhook signature — the endpoint is public; an attacker could POST a fake “payment succeeded.”
Process the webhook → update the payment row → emit the internal event. Treat the webhook as another path that resolves a pending row, alongside the synchronous response and reconciliation.

7. Downstream — don’t dual-write

After a payment succeeds you must tell other services (fulfil the order, send a receipt). Doing UPDATE payment then publish event as two separate steps is the dual-write problem — a crash between them loses the event.

Use the transactional outbox: in the same DB transaction that marks the payment succeeded, insert an outbox row for the payment.succeeded event. A relay process publishes outbox rows to the queue. Downstream consumers are idempotent (keyed on payment id). Now the payment status and the event are atomic — both happen or neither.

8. State machine

Model the payment as an explicit state machine — pending → succeeded, pending → failed, and only those transitions. No succeeded → pending. Every transition is a recorded, audited row change. This makes the system reason-about-able and is itself an interview talking point: “I’d model it as an explicit state machine so illegal transitions are impossible and every change is auditable.”

9. Bottlenecks & trade-offs

Correctness over latency — the synchronous flow is deliberately not the fastest path; the client waits for a real answer or a clear “in progress.” That’s correct for payments — never trade correctness for latency here.
The timeout ambiguity is unavoidable — a timed-out processor call is genuinely “unknown”; the system’s job is to resolve the unknown safely (query by key, reconcile), never to guess (blind retry).
Idempotency key TTL — keep keys long enough that any legitimate retry window is covered (hours to days); they’re cheap to keep.
Scale — payments QPS is usually modest; this rarely needs sharding. If it did, shard payments by idempotency key — every operation for one payment is keyed by it.
Don’t roll your own processor integration loosely — lean on the processor’s idempotency support and its query-by-key API; they exist precisely for this.

Interview angle

“How do you prevent double-charging?” — a client-generated idempotency key per payment intent, stored with a unique constraint: INSERT ... ON CONFLICT DO NOTHING. A retry hits the existing row — return the stored response, don’t charge again. Pass the same key to the processor so it dedups too.
“The call to Stripe times out — you don’t know if it charged. What do you do?” — do not blindly retry (double-charge risk). Leave the row pending and resolve the unknown: query the processor by the idempotency key — “did a charge with key X succeed?” — and reconcile the row from the answer.
“How do you make sure a payment is never lost?” — a reconciliation job. Idempotency keys stop double-charges; reconciliation stops lost payments. It sweeps stale pending rows, queries the processor by key, and converges your DB to the processor’s reality. A pending row is always a question something must answer.
“How do you handle the payment webhook?” — verify the signature (public endpoint), dedup on the processor’s event id (webhooks are at-least-once), then treat it as another path that resolves the payment row — alongside the sync response and reconciliation.
“How do you tell downstream services without losing the event?” — transactional outbox: insert the payment.succeeded event into an outbox table in the same transaction as the status update. A relay publishes it; consumers are idempotent. Avoids the dual-write problem (crash between DB update and publish loses the event).
“Why model it as a state machine?” — explicit pending → succeeded/failed transitions make illegal states impossible, every change is an audited row, and the whole flow becomes reason-about-able. Payments demand auditability and correctness over cleverness.