backendgym backendgym · projects Sign in with GitHubSign in

projects /09-money-from-nowhere

# The wallet that spent the same credit twice

One Hearth customer spent the same wallet credit twice and ended up below zero. Reproduce the negative balance, work out whether the slow PSP is involved, and restore the invariant.

Advanced ~5h PythonFastAPIPostgreSQL

practises transactions, bulkheads, isolation, locking, race-detection, fault-injection

prepare with SQL · Querying, SQL · Designing and writing, SQL · Viewing

Sign in with GitHub
money-from-nowhere / incident.md v1.0
Symptoms

10:17, Tuesday

The wallet alert fires: one balance has gone negative. Soon after, a customer, Cora-17, tells support that a 75.00 credit purchase went through twice, although the wallet only held 100.00. The API returned success both times.

Hearth is a marketplace, and customers spend wallet credit at checkout through POST /buy. A balance below zero is a ledger problem, and here it means Hearth handed over 50.00 of goods nobody paid for.

There's more on the page. The PSP team reports a slow stretch: charges that normally take 150ms locally took 500ms during the busiest minute, though they found no duplicate charge ids. A deploy to the wallet service went out 18 minutes before the alert. Wallet API errors are steady at 0.2%, and Postgres is taking connections.

You have 30 minutes to reproduce and contain it. Don't drop the wallet or purchase tables, don't reset the whole database, and don't restart the service in place of a fix.

What's running

The starter has the wallet API, its Postgres tables, a fake PSP whose latency you control, request evidence, lock-wait metrics, and a load command that sends the same two purchases every time. The customers and the payment gateway are local, and two requests stand in for concurrent traffic. PSP settlement reconciliation, idempotency across regions and the production ledger migration plan go in the postmortem.

starter Tserewara/starter-money-from-nowhere · you generate your copy when you start

Requirements

What has to be true when you're done. How you get there is up to you.

  1. R1
    After two concurrent purchases on the same wallet, no balance is negative and no more credit has been spent than the wallet held.
    ask the stakeholder

    start the project to ask

  2. R2
    The fix leaves operators an invariant check they can see, and a metric that shows lock waiting before it turns into a customer's balance problem.
    ask the stakeholder

    start the project to ask

Questions for the stakeholder

They reveal no requirement, but the answers shape the design.

start the project to ask

Runbook

Reproduce, diagnose, fix and verify, in that order.

  • D1 reproduce R1 , not recorded yet

    Reproduce the negative balance: run make load, which resets Cora-17's wallet and sends two concurrent purchases, then read make evidence.

    expectThe load prints "statuses": [200, 200] and "negative_balances": 1.

  • D2 diagnose , not recorded yet

    Slow the PSP to 500ms with make psp-slow, run make load again, and compare the PSP latency with the balance_read and wallet_lock events in make evidence.

    expectThe PSP is slower, but the extra spend comes from two requests reading one balance. Nothing in the evidence shows the PSP creating it.

  • D3 fix R1 , not recorded yet

    Make the balance check and the debit atomic under concurrency, reconcile the negative row D1 and D2 left behind (before you run D4, since make load resets the wallet), and keep a purchase record support can use.

    expectAfter the data change the wallet invariant holds, and a concurrent purchase the wallet can't fund is rejected without another debit.

  • D4 verify R1 , not recorded yet

    Once D3's reconciliation is recorded (the load resets Cora-17's wallet), run make load against the fixed service, then check make metrics.

    expectThe load prints one 200 and one 409, negative_balances is 0, purchase_count is exactly 1, and lock_wait_p99_ms is still reported.

Technical guidance

  • ▸Build the evidence trail before you touch the transaction. Every timing you see is data; decide what it explains once the trail is complete.
  • ▸For each request, compare the balance it read, the balance it locked and the balance it committed.
  • ▸Two successful responses don't settle anything. Reconcile the wallet and purchase records after the load.

Done when

Every drill above passes, and:

  • The postmortem covers the timeline, the mechanism, what the fix cost, and the check that would have caught it.
  • A data change reconciles the negative balance the reproduction leaves behind, without dropping either table.
Solutions

No solutions published yet. Ship the project and yours will be the first.