The orders that never reached the warehouse

The incident

BrasaCart ships about 420 orders a minute from its checkout API to a warehouse picking system. The warehouse feed is part of the carrier cutoff contract: a paid order should be visible there within five minutes.

On Tuesday at 09:12, the message broker stopped accepting connections for 90 seconds. Checkout still accepted 612 orders and their rows were present when the broker returned, but 37 of those orders never appeared in the warehouse feed. The late orders missed the first pick wave, and support refunded 11 customers who thought BrasaCart had lost their purchases. A separate replay during recovery produced two picking records for one order.

Build a handoff that survives the broker pause, can be safely delivered again, and isolates a message the warehouse cannot read. The warehouse API and multi-region operations are outside this lab.

Lab boundary

The API, SQLite order store, RabbitMQ broker, and warehouse worker run locally. A paused RabbitMQ container simulates the 90-second broker outage; the included poison-message control simulates a malformed event. You will implement the durable handoff, relay, consumer guard, and dead-letter route. Multi-node fencing, cross-region recovery, and warehouse reconciliation policy belong in the write-up.

Starter

Use the template at https://github.com/Tserewara/starter-the-order-that-never-shipped.

P1 · Keep the committed order visible

Close the gap between a successful checkout and a broker that is not accepting work.

R1 Every order committed while the broker is unavailable eventually produces a warehouse event when the broker returns.

R4 The order write and the record that makes its event eligible for delivery share one database transaction.

Failure drills

D1Pause RabbitMQ, create 100 orders through the API, resume the broker, and compare committed orders with warehouse side effects after recovery.
D2Inspect the database transaction and force the broker connection to fail immediately after an order request begins; then find the durable record the relay can use.

○The service has a durable handoff record that a recovery process can discover.

P2 · Make delivery safe to repeat

Survive consumer crashes, malformed messages, and the retries that follow them.

R2 A repeated delivery of one event id creates at most one warehouse side effect.

R3 hidden until you ask

Failure drills

D3Publish the included malformed event, then publish a valid order and inspect the queue and worker output while the malformed delivery is retried.
D4Stop the worker during a delivery, start it again, and inspect the warehouse-side effect records for the event that was in flight.
D5Send 1,000 deliveries with worker restarts and redeliver the same event ids, then compare unique event ids with warehouse side effects.

○The consumer and queue policy make a repeated or unreadable delivery an explicit state rather than an outage.

Technical guidance

▸Start with the two durable facts: what the API commits and what the warehouse has acknowledged.

▸Treat a broker outage and a consumer redelivery as different failures; one loses visibility and the other repeats work.

▸Make the retry and replay policy visible in the queue declarations and in the event records.

▸Run the same load after the change. A design that sounds durable but cannot show its counts is not finished.

Definition of done

All of them, or it isn't shipped.

lost order events under 0 events at the same load D1

The order row and its delivery record commit together, or neither commits; a broker failure cannot leave a committed order with no durable delivery record. D2

The malformed event reaches a dead-letter queue after the configured retry bound, and the valid order is processed afterward. D3

A redelivery is harmless: the event may be handled again, but the warehouse records one side effect for its event id. D4

duplicate warehouse side effects under 0 per 1000 events at the same load D5

The API, relay, worker, queue declarations, and database changes are versioned in the repository.

A clean clone documents one command to start the stack and a safe operator path for inspecting and replaying quarantined events.

The write-up records the chosen delivery and deduplication trade-offs, including what remains a multi-region design question.

nothing recorded yet — running the drills fills these in