backendgym backendgym — projects Sign in with GitHubSign in

projects /05-twice-the-email

# Twice the shipping email

A worker restart made RotaSul send the same shipping notice twice; trace the delivery, fix the invariant, and prove the count is zero.

Intermediate ~10h PythonRabbitMQDocker
Sign in with GitHub
twice-the-email / incident.md v1.0
Symptoms

The on-call page

RotaSul sends about 2,800 shipping emails an hour from its fulfillment queue. At 16:40, support started forwarding customer screenshots: the same shipment notice had arrived twice, with the same tracking number. The 18 tickets all followed a restart of the shipping-email worker during a routine host patch.

The provider dashboard showed 99.9% accepted requests and a p95 response time of 110ms. The publisher had accepted the events, the queue was reachable, and no other notification type was duplicated. The provider team says it does not retry a request after returning success.

You have 45 minutes to work the page. Do not delete the queue or change the provider. You may restart the worker, inspect its logs and the provider request log, and change the consumer.

Lab boundary

The publisher, RabbitMQ, worker, and email provider run locally. A controlled worker restart simulates the host patch; the provider is a deterministic local stand-in that records every accepted request. Production mail reputation, provider failover, and multi-region queue recovery are outside this incident.

Requirements

What must be true — the how is yours.

  1. R1 One shipping event id produces at most one accepted email side effect, including when the worker restarts after the provider accepts the request. rehearsed by D1 · D4 · D5

Technical guidance

  • ▸Build the timeline from the worker log, broker delivery, provider request, and acknowledgement rather than from the customer screenshot alone.
  • ▸A successful provider response tells you the provider accepted a request; it does not tell you the consumer can safely acknowledge the event.
  • ▸Keep the fix at the event boundary and name the invariant it restores.

Definition of done

All of them, or it isn't shipped.

  • The same event id appears twice in the provider log after the restart, with two distinct provider request ids. — not recorded yet D1
  • The provider accepted two separate requests and returned success to each; its log contains no retry action, so provider retrying is ruled out. — not recorded yet D2
  • The event id is redelivered after the worker dies before acknowledgement, and the consumer has no durable event-id guard before performing the side effect. — not recorded yet D3
  • The consumer restores the one-email-per-event invariant using durable state; a redelivery is acknowledged or safely skipped without a second provider request. — not recorded yet D4
  • duplicate emails under 0 per 1000 events at the same load — not recorded yet D5
  • The postmortem contains a timeline, root cause, customer impact, corrective change, and a guard that would have caught the failure. — not recorded yet
  • The fix remains safe across a fresh worker process and does not depend on an in-memory set. — not recorded yet

nothing recorded yet — running the drills fills these in

spec v1.0 · drafted with AI, human-reviewed

Solutions

No solutions published yet. Complete the project and yours opens the thread.