Twice the shipping email

Symptoms

16:40

Support starts forwarding screenshots. Customers are getting the same shipping notice twice, same tracking number both times. SouthRoute sends about 2,800 of these an hour from its fulfillment queue, and all 18 tickets so far came in after the shipping-email worker restarted during a routine host patch.

Nothing else looks wrong. The email provider's dashboard shows 99.9% of requests accepted and a p95 of 110ms. The publisher accepted the events, the queue is reachable, and no other kind of notification is duplicated.

You have 45 minutes. You can restart the worker, read its logs and the provider's request log, and change the consumer. Don't delete the queue, and don't change the provider.

What's running

The publisher, RabbitMQ, the worker and the email provider all run locally. You rehearse the host patch by crashing the worker on purpose, and the provider is a predictable local stand-in that records every request it accepts. Mail reputation, provider failover and multi-region queue recovery are out of scope for this one.

Requirements

What has to be true when you're done. How you get there is up to you.

One shipping event id causes at most one accepted email, including across worker restarts.

Questions for the stakeholder

They reveal no requirement, but the answers shape the design.

start the project to ask

Runbook

Reproduce, diagnose, fix and verify, in that order.

D1 reproduce R1

Reproduce the double send: run make arm-crash, then POST http://localhost:8001/shipments with {"email": "ana@example.test"} (it returns the event_id). When the worker exits, start it again (COMPOSE_PROJECT_NAME=bgym_email docker compose start worker) and run make stats.

expectAfter the restart the provider log has the same event id twice.

D2 diagnose

Put the two provider log entries next to the provider's responses and the worker log. Did the provider retry, or did the worker send two requests?

expectThe provider received two separate requests and returned success for both. Nothing in its log is a retry, so the provider is ruled out.

D3 diagnose

Read the queue delivery and the worker timeline around the restart, and find what lets one event reach the provider twice.

expectThe worker dies after the send and before the acknowledgement, so the event is redelivered, and nothing durable in the consumer checks the event id before sending.

D4 fix R1

Change the consumer and start it as a clean process. Run make arm-crash, create one shipment, start the worker again when it exits, then replay that event with POST http://localhost:8001/admin/replay/<event_id> and check make stats.

expectThe provider has one request for that event id. The redelivery and the replay are acknowledged or skipped, and the consumer's record of sent events is durable, so a fresh process still sees it.

D5 verify R1

From a clean stack (make down, then make up), run make arm-crash, then make duplicate-load (1,000 shipments, then a replay of the first 20 event ids). When it says the worker is not running, start it again; the script waits for the queue to drain before printing.

expectThe line reads provider_requests=1000 duplicate_side_effects=0: no duplicate emails for 1,000 events, even across the crash and the replays.

Technical guidance

▸Build the timeline from the worker log, the broker delivery, the provider request and the acknowledgement. Start from the customer's screenshot, then check each component's record.

▸Treat each component's success as a claim, and check it against what the other components recorded.

▸Fix it at the event boundary, and say which invariant the fix restores.

Done when

Every drill above passes, and:

The postmortem has a timeline, the root cause, the customer impact, the change you made, and a check that would have caught it.

The fix still holds after the worker is replaced by a fresh process.