The notification queue that made OTPs wait

Last month

Bellbird is a marketplace. When a buyer's order changes, it sends push, email, SMS and in-app notifications. One-time passcodes go out through the same system, and a login code is only useful for a short while. A newsletter can wait.

Last month a newsletter to 25,000 people was sitting in the queue when a login campaign sent 1,800 OTPs. The first code arrived 19 seconds after it was requested, and 14% of them arrived too late to use. Then a provider went down, and afterwards nobody could say what had happened to 640 notifications.

Build a delivery path an operator can run: urgent messages don't wait behind bulk mail, and provider trouble doesn't erase work that was already accepted. Provider contracts, campaign authoring and the mobile client aren't part of this.

What the starter gives you

The notification service (Python and FastAPI): an API that accepts notifications into a Redis queue, and one worker in the same container that hands them to the provider in order. Around it, a fake provider that stands in for every vendor: you can slow it down or take it down, and it keeps every notification it accepts with the time it arrived. make up starts it all.

make notify replays last month on a small scale: fifteen newsletters, then one urgent login code, and one line saying how long the code took to reach the provider. make outage takes the provider down while five notifications are accepted, brings it back two seconds later, and counts what arrived, what the service gave up on and what was lost; make outage-long does the same with ten seconds of outage; whether those five end delivered or given up on depends on the retry budget you choose, and either counts as accounted for. make contract checks the service still answers as it did.

You can port the service to another language: keep its routes, port 8000 inside the container, REDIS_URL and PROVIDER_URL, and run make contract until it passes.

Per-vendor quotas, a queue that spans regions and user preferences don't fit on a laptop. Cover them in your design notes.

The work, in 3 phases

Each phase says what must be true and the drill that proves it. You close a phase when its drills pass.

P1 · Deliver one notification

Check the path as it stands: a notification is accepted, handed to the provider and recorded.

R1
In the normal case, accepted notifications reach their provider, and the service records a final outcome for every one.

D1 R1 , not recorded yet
With the stack up, POST /notifications with {"channel": "email", "body": "hello"}, poll GET /notifications/{id} until it settles, and look for it with make received.

expectThe notification comes back delivered, and the provider has it under the same id.

done when`make contract` still passes after your change.

P2 · Make urgent work move

A login code sits behind a newsletter backlog. Measure how long it waits, change the worker, measure again.

R2
With newsletters queued, an urgent login code no longer waits behind them: it reaches the provider ahead of the backlog, fast enough for someone watching a login screen.

D2 R2 , not recorded yet
With the provider at its normal speed, run make notify before changing anything and note otp_ms. After your change, run it again.

expectAll sixteen are delivered both times, and after your change the login code reaches the provider ahead of the backlog. Say how far otp_ms fell and what the bulk work gave up for it.

P3 · Contain provider trouble

The provider fails while work is being accepted, and nothing accepted may disappear.

R3
A provider outage loses none of the accepted notifications: each one is delivered, or given up on after a bounded number of attempts and kept where an operator can see it.

D3 R3 , not recorded yet
Run make outage, then make outage-long, and read delivered, dead_lettered and lost on each line. Look up one given-up notification with GET /notifications/{id}.

expectlost is 0 both times. Every notification is delivered or dead_lettered with its error, and retries stop after the number of attempts your design sets.

Questions for the stakeholder

They reveal no requirement, but the answers shape the design.

start the project to ask

Technical guidance

▸Keep the load fixed while you measure: `make notify` sends the same backlog every time.

▸Decide which state is durable before you pick a retry count. A retry that leaves no outcome record is hard to operate.

▸The service reports a notification it gave up on as `dead_lettered`; `make outage` counts that as accounted for.

# The notification queue that made OTPs wait