The wallet that spent the same credit twice

Symptoms

The on-call page

Hearth is a marketplace with a wallet endpoint, POST /buy. Customers use wallet credit at checkout, so a balance below zero is both a ledger problem and a promise the company cannot keep.

At 10:17 on Tuesday, the wallet alert reported one negative balance. Cora-17 then told support that one 75.00 credit purchase appeared twice, even though the account had only 100.00. The API returned two successful responses. The PSP team also reported a slow period: charge latency was 150ms locally and 500ms during the busiest minute. Wallet API errors stayed at 0.2%, and Postgres was accepting connections.

A deploy to the wallet service happened 18 minutes before the alert. The PSP was slow, but its team found no duplicate charge ids. You have 30 minutes to reproduce and contain the issue. You may not drop the wallet or purchase tables, reset the whole database, or restart the service as a substitute for a fix.

Lab boundary

Implemented: a wallet API, Postgres tables, a fake PSP with controllable latency, request evidence, lock-wait metrics, and a deterministic two-buyer load command. Simulated: the customer load and payment gateway are local, and the two requests stand in for concurrent traffic. Design doc: PSP settlement reconciliation, idempotency across regions, and the production ledger migration plan.

Starter

Use the template at https://github.com/Tserewara/starter-money-from-nowhere.

Requirements

What must be true — the how is yours.

R1 After two concurrent purchases for the same wallet, no wallet balance is negative and no more credit is spent than the wallet held. rehearsed by D1 · D3 · D4

R2 The fix leaves an operator-visible invariant check and a metric that exposes lock waiting before it becomes a customer balance incident.

Technical guidance

▸Build the evidence trail before changing the transaction; the slow PSP is real data, not a conclusion.

▸Compare the balance each request observed with the balance it locked and the balance it committed.

▸A successful response is not enough: reconcile the wallet and purchase records after the load.

Definition of done

All of them, or it isn't shipped.

The first run produces two successful purchase attempts and a negative wallet balance; the evidence includes two request ids that observed the same starting balance. D1

The PSP is slower, but the duplicate outcome is explained by two requests observing one balance; the evidence does not blame the PSP for creating the extra spend. D2

The wallet invariant holds after the data change and a concurrent attempt that cannot be funded is rejected without another debit. D3

negative wallet balances after load under 0 wallets at the same load D4

The postmortem names the timeline, the mechanism, the fix cost, and the guard that would have caught the issue.

The data change reconciles the seeded negative balance without dropping either table.