backendgym backendgym — projects Sign in with GitHubSign in

projects /08-notify

# The notification queue that made OTPs wait

Bellbird's newsletter backlog made one-time codes wait while providers failed and throttled; separate urgency from delivery risk and prove the result.

Advanced ~28h PythonFastAPIRedis phase 1 of 4 · Deliver one notification
Sign in with GitHub
notify / spec.md v1.0

The incident

Bellbird is a fictional marketplace. It sends push, email, SMS, and in-app notifications when a buyer's order changes. One-time passcodes are part of sign-in and need a quick answer; newsletters can wait.

Last month, a 25,000-recipient newsletter filled the notification queue when a login campaign sent 1,800 OTPs. The first OTP arrived 19 seconds after the request, 14% of the OTP requests missed their useful window, and a provider outage left 640 notifications unaccounted for. The providers also enforce separate rate limits, and some users have opted out of marketing channels.

Build the delivery path the support team can operate: urgent messages should not sit behind bulk mail, provider trouble should not erase accepted work, and a user's choices must be respected. The lab does not include provider contracts, campaign authoring, or a mobile client.

Lab boundary

Implemented: a notification API, Redis-backed queue and records, one worker, a fake provider with controllable latency and failure, a load script, and delivery metrics. Simulated: provider rate limits are represented by the fake provider's delay and outage controls, and the 25,000-recipient campaign is represented by a smaller deterministic queue. Design doc: provider-specific quotas at fleet scale, multi-region queue placement, push-token lifecycle, and the client protocol details beyond the local SSE stream.

Starter

Use the template at https://github.com/Tserewara/starter-notify.

P1 · Deliver one notification

Get a normal notification through the queue and worker before adding policy.

  1. R1 Accepted notifications reach their selected provider in the normal case, and the worker records a final outcome for each one.
Failure drills
  • D1Start the stack and queue one email notification, then poll its status and inspect the delivery metrics.
  • ○The local API, worker, Redis, and provider are declared and the normal delivery is repeatable from a clean clone.

P2 · Make urgent work move

Put a one-time code behind a newsletter backlog and measure the wait.

  1. R2 An urgent OTP has p95 delivery time at or below 250ms in the shipped priority-queue drill while newsletters are queued.
Failure drills
  • D2Queue fifteen newsletters followed by one urgent push notification with the supplied load command; record the OTP p95 before changing the worker.
  • ○The before and after runs use the same bulk count, urgent message, provider latency, and metric output.

P3 · Contain provider trouble

Keep one provider's failure from erasing accepted work or blocking its neighbors.

  1. R3 A provider outage loses zero accepted notifications after bounded retries and dead-letter handling are in place.
Failure drills
  • D3Make the provider fail, queue five notifications, restore the provider during the retry window, and inspect final outcomes and the dead-letter list.
  • ○Retry and dead-letter policy is visible in stored outcomes and can be explained after the outage run.

P4 · Respect the person receiving it

Add preference-aware delivery and an in-app path that can reach a live client.

  1. R4 hidden until you ask
Failure drills
  • D4Set a user's marketing preference to opted out, send a newsletter and an OTP, then open the in-app event stream for an opted-in user and inspect the delivery counts.
  • ○The preference and stream behavior are documented for reconnects and for transactional messages.

Technical guidance

  • ▸Keep provider calls behind one boundary so latency, failure, and rate limits can be changed during a run.
  • ▸Give the measurement a fixed queue shape: bulk work first, then an OTP, with the same provider latency before and after.
  • ▸Decide which state is durable before choosing a retry count; a retry without an outcome record is hard to operate.

Definition of done

All of them, or it isn't shipped.

  • delivered notifications over 1 notifications at the same load — not recorded yet D1
  • The FIFO starter delivers all notifications, but the OTP waits behind the newsletters; record its baseline p95. — not recorded yet D2
  • accepted notifications lost under 0 notifications at the same load — not recorded yet D3
  • opted-out marketing deliveries under 0 notifications at the same load — not recorded yet D4
  • A clean clone starts the API, worker, Redis, and fake provider with one command and the README describes the controls used by the drills. — not recorded yet
  • The final delivery policy records ordering, retry, rate-limit, preference, and client-stream trade-offs in the design notes. — not recorded yet

nothing recorded yet — running the drills fills these in

spec v1.0 · drafted with AI, human-reviewed

Solutions

No solutions published yet. Complete the project and yours opens the thread.