The orders that never reached the warehouse

Tuesday, 09:12

EmberCart's checkout API sends about 420 orders a minute to the warehouse picking system. The carrier cutoff contract says a paid order has to show up in the warehouse feed within five minutes.

At 09:12 on Tuesday the message broker stopped accepting connections for 90 seconds. Checkout kept going and accepted 612 orders, and all of those rows were in the database when the broker came back. 37 of them never reached the warehouse. They missed the first pick wave, and support refunded 11 customers who thought their purchase had been lost. A replay during recovery also left one order with two picking records.

What you're changing

The handoff from checkout to the warehouse has to survive a broker pause and be safe to deliver twice. The warehouse API and running this across regions are out of scope.

What the starter gives you

Checkout (Python and Flask) with its SQLite order store, RabbitMQ, and the worker that turns each order event into a row in the warehouse's picking feed: a picks table in a Postgres database (WAREHOUSE_URL) that the harness counts. You may add tables or constraints to that database; a row in picks still means one box packed. make up starts it all; the API is on localhost:8000.

make outage replays Tuesday on a small scale: it pauses the broker, sends 100 orders, brings the broker back and, once the feed settles, prints how many of those orders reached the warehouse, how many were lost and how many were picked twice. make deliveries sends 1,000 orders, replays every one of their events the way operators did during recovery, and prints the same counts; restart the worker with make stop-worker and make start-worker while it runs. make picks counts the feed at any time, and make contract checks checkout still answers as it did.

You can port the service to another language: one image that runs as api and as worker, the same routes, DB_PATH, RABBITMQ_HOST and WAREHOUSE_URL, and make contract until it passes. A process you add runs inside one of those two roles.

Fencing across nodes, recovery across regions and reconciling with the warehouse go in your README.

The work, in 2 phases

Each phase says what must be true and the drill that proves it. You close a phase when its drills pass.

P1 · Don't lose the committed order

The broker pauses while orders come in. Every committed order has to reach the warehouse anyway.

R1
Every order committed while the broker is down reaches the warehouse feed once the broker is back.
R2
No committed order is left without a pending delivery, whatever the broker does at the moment of the commit.
R3
hidden until you ask the stakeholder
ask the stakeholder
start the project to ask

D1 R1 , not recorded yet
Run make outage before changing anything, then again after your change, and read lost on the last line each time.

expectAfter your change, every order accepted during the pause is in the warehouse and lost is 0.
D2 R2 , not recorded yet
Read the code path that commits an order, then make pause-broker, create one order with POST /orders, and find what, if anything, still records that it is owed to the warehouse. make resume-broker when you are done.

expectYou can point to the row, written in the same transaction as the order, that says the order is still owed to the warehouse, and it is still there while the broker is paused.

done when`make contract` still passes after your change.

P2 · Make delivery safe to repeat

Replays and redeliveries happen. The warehouse must still pick each order once.

R4
Delivering the same event again, by a replay or a redelivery after a crash, adds no second pick for its order.

D3 R4 , not recorded yet
Run make deliveries before changing anything, then again after your change, restarting the worker a few times while it runs (make stop-worker, make start-worker). Read duplicates on the last line.

expectAfter your change duplicates is 0 and every one of the 1,000 orders has one pick.

done when`make contract` still passes after your change.

Questions for the stakeholder

They reveal no requirement, but the answers shape the design.

start the project to ask

Technical guidance

▸Write down the two things that have to stay true: every committed order reaches the feed, and no order is picked twice.

▸Draw where each piece of state lives (the order store, the broker, the feed) before you add a table.

Done when

Every drill above passes, and:

The README records the delivery and deduplication trade-offs you chose, and what stays open: fencing across nodes, other regions, reconciling with the warehouse.

# The orders that never reached the warehouse