backendgym backendgym — projects Sign in with GitHubSign in

projects /07-cold-cache-checkout

# The checkout prices that vanished after a deploy

A deploy emptied Mira's price cache and a burst of shoppers overwhelmed the store; make cold reads safe, measurable, and useful during a Redis outage.

Advanced ~24h PythonFastAPIRedisPostgreSQL phase 1 of 4 · Make the price path work
Sign in with GitHub
cold-cache-checkout / spec.md v1.0

The incident

Mira is a fictional price-comparison marketplace. Its checkout page calls GET /price/{sku} for every item in the basket, and that lookup serves 71% of requests. Prices change often enough that the team keeps them in Redis in front of Postgres.

At 14:02 on Thursday, a deploy flushed Redis. Forty thousand product pages then missed the cache within two minutes. Postgres reached 96% CPU, price p99 rose from 11ms to 2.8s, and completed checkouts fell 31% for nine minutes. The store recovered when the traffic spread back across warm keys.

The action item is to make this read path survive a cold cache and a missing Redis dependency. Keep the prices correct and keep the checkout useful; changing the pricing rules or moving the database is outside the job.

Lab boundary

Implemented: one FastAPI price endpoint, Redis cache-aside behavior, Postgres reads, a burst load generator, counters, and controls for expiring a hot key or simulating a dependency outage. Simulated: the 40,000-page burst is represented by concurrent local requests, and Postgres size is represented by the same indexed lookup at a smaller scale. Design doc: multi-region caching, cache warming across a fleet, and the operational policy for prices older than the local 30-second window.

Starter

Use the template at https://github.com/Tserewara/starter-cold-cache-checkout.

P1 · Make the price path work

Establish the warm and cold behavior before the load exposes its cost.

  1. R1 A cold request returns the current price, and a warm request does not read Postgres again.
Failure drills
  • D1Clear the hot key, request `sku-1` once, then request it again and inspect the response source and store-read counter.
  • ○The local API, Redis, and Postgres are declared and the happy path is repeatable from a clean clone.

P2 · Put a crowd on the miss

Use one hot key to show what a naive miss does to the store.

  1. R2 A burst of misses for one SKU causes a bounded number of Postgres reads rather than one read per request.
Failure drills
  • D2Expire `sku-1`, send a 40-request concurrent burst for that SKU, and record the store-read counter and p99.
  • ○The baseline run is recorded with the same request count and hot key that the evolved run will use.

P3 · Choose the shape of recovery

Change the read path, then measure the same burst against the baseline.

  1. R3 The evolved read path keeps the burst p99 below 250ms in the shipped local drill.
Failure drills
  • D3Run the same expired-key burst after your coordination or refresh change, using the starter's JSON output to compare store reads and p99 with D2.
  • ○The design choice and its failed-refresh behavior are documented beside the implementation.

P4 · Keep checkout useful without Redis

Let the business promise decide what a dependency outage looks like.

  1. R4 hidden until you ask
Failure drills
  • D4Warm `sku-1`, make Redis unavailable and then make the store unavailable, and request the SKU.
  • ○The outage controls and freshness policy are documented for the next on-call.

Technical guidance

  • ▸Measure cold and warm requests separately; one average hides the failure that matters.
  • ▸Keep the burst on one hot key so store-read counts describe coordination rather than cache-key diversity.
  • ▸Write down what freshness a shopper can tolerate before choosing between waiting, serving stale data, and refreshing early.

Definition of done

All of them, or it isn't shipped.

  • store reads for two requests under 1 reads at the same load — not recorded yet D1
  • The naive path returns the price to every caller, while the store sees a read for each concurrent miss; record that baseline before changing the design. — not recorded yet D2
  • burst p99 latency under 250 ms at the same load — not recorded yet D3
  • stale prices served during outage over 1 responses at the same load — not recorded yet D4
  • A clean clone starts the API, Redis, and Postgres with one command and the repository documents the controls used by the drills. — not recorded yet
  • The final choice records its freshness, failure, and load trade-offs in the design notes. — not recorded yet

nothing recorded yet — running the drills fills these in

spec v1.0 · drafted with AI, human-reviewed

Solutions

No solutions published yet. Complete the project and yours opens the thread.