The payment that happened twice

Friday afternoon

Ledgerloop sells subscriptions. POST /v1/charges is the last call in its checkout, and about 18% of the month's revenue goes through it.

On Friday a mobile carrier dropped responses for eleven seconds. The apps retried, as apps do: 240 charge requests went out a second time, and 17 customers were charged twice. Support refunded $1,860 that afternoon. Finance held the settlement export for two hours while they worked out which charges were real.

The API took every retry at face value. Nothing in a request tells it "I've already been here", so a retry and a new payment look the same.

What you're changing

Make the endpoint safe to retry. The request and response shapes stay as they are, and so does the charge path behind them; clients depend on both. Leave the payment provider alone, and don't build a durable multi-region ledger here.

What the starter gives you

The API runs locally with an in-memory charge store. make load replays Friday on a small scale: forty concurrent attempts at one checkout, then a line saying how many charges came out of it. Run it once before you change anything.

The local setup only measures what happens at the request boundary. Where the idempotency record would live in production, what happens to it on a restart or a second instance, and how you'd reconcile with the provider go in your README.

Requirements

What has to be true when you're done. How you get there is up to you.

Sending the same idempotency key with the same body again returns the first charge's result and creates no new charge.

hidden until you ask the stakeholder

ask the stakeholder

start the project to ask

Failure drills

Break the system on purpose, then record what happened on VERIFY.

D1 R1

Send one charge body twice with the same idempotency key. Compare the two responses, then list what was stored with GET /charges, the local inspection endpoint.

expectBoth responses describe the same charge, and GET /charges lists it once.

D2 R1

After your change, run make load (forty concurrent attempts with one key) and read the summary line it prints.

expectThe summary ends with charges=1 duplicate_charges=0.

D3 R2

Send a charge with a key, then send a different amount under the same key before the key expires.

expectThe second request gets a conflict, and the first charge is still the only one for that key.

D4 R2

Create a charge, move the service clock forward 31 seconds (POST /_control/advance with X-Advance-Seconds: 31), then retry the key with a different amount.

expectThe retry creates a new charge, and the old charge is still recorded.

Technical guidance

▸The key and the body are one contract. A replay is safe only when the stored request matches the new one.

▸Checking the key and creating the charge have to be a single atomic decision, even with the in-memory store.

▸Get the behavior right first, then run the storm. Two sequential calls passing tells you little about forty concurrent ones.

Done when

Every drill above passes, and:

The starter tests still pass, so responses and validation behave as before.

The README says where idempotency records would live in a real deployment, how long they last, and why.