projects /07-cold-cache-checkout

# The checkout prices that vanished after a deploy

A deploy flushed Mira's price cache and the checkout fell over on the rush of misses that followed. Make cold reads safe, measure them, and keep checkout usable when Redis goes away.

Advanced ~5h suggested:PythonFastAPI RedisPostgreSQL 4 phases

practises caching, cache-stampede, postgresql, graceful-degradation, fault-injection, percentiles

prepare with SQL · Querying, SQL · Designing and writing, SQL · Viewing

cold-cache-checkout / spec.md v1.0

Thursday, 14:02

Mira compares prices. Its checkout calls GET /price/{sku} for every item in the basket, and that one lookup is 71% of the traffic. Prices are read far more often than they change, so they sit in Redis for 30 seconds in front of Postgres.

At 14:02 on Thursday a deploy flushed Redis. Within two minutes forty thousand baskets had missed the cache and gone straight to Postgres. The database hit 96% CPU, price p99 went from 11ms to 2.8s, and completed checkouts dropped 31% for nine minutes. It recovered on its own once traffic spread back over keys that were warm again.

The action item from the review: this read path has to survive a cold cache, and it has to survive Redis not being there at all. Prices stay correct and the checkout stays usable. The pricing rules and the database stay where they are.

What the starter gives you

The price service (Python and FastAPI) doing plain cache-aside over Redis and Postgres, and a harness around it. make up starts it on localhost:58000; every answer says where the price came from in source. make burst replays Thursday on a small scale: it empties Redis the way the deploy did, sends 40 requests for one SKU at the same moment, standing in for the forty thousand baskets, and prints how many reads reached Postgres and the p99 the callers saw. Postgres counts those reads itself, and it is tuned to behave like a busier primary than a laptop's. make redis-down and make store-down stop Redis or Postgres for real, and make contract checks the service still answers as it did.

You can port the service to another language: keep its route, port 8000 inside the container, DATABASE_URL and REDIS_URL, and run make contract until it passes.

Some of this won't fit on a laptop: caching across regions, warming the cache across a fleet, and what the business does with a price fetched from Postgres more than 30 seconds ago. Put those in your design notes.

starter Tserewara/starter-cold-cache-checkout · you generate your copy when you start

The work, in 4 phases

Each phase says what must be true and the drill that proves it. You close a phase when its drills pass.

P1 · Check the price path

The starter already does cache-aside. Confirm a warm read skips Postgres before you put load on it.

R1
A cold request returns the current price, and a warm request for the same SKU doesn't touch Postgres.

D1 R1 , not recorded yet
make flush, then request sku-1 twice and run make reads before and after. Look at each response's source.

expectThe first response has "source": "store", the second "source": "cache", and store_reads went up by 1.

done when`make contract` still passes after your change.

P2 · Put a crowd on the miss

Point forty callers at one hot key on an empty cache and record what happens.

R2
A burst of misses for one SKU causes a bounded number of Postgres reads, and you can say what bounds it.

D2 R2 , not recorded yet
Before changing anything, run make burst and write down store_reads and p99_ms from the line it prints.

expectEvery caller gets the price, and you have the baseline line: store_reads and p99_ms before any change.

P3 · Choose how it recovers

Change the read path, then run the same burst and compare with the baseline.

R3
After your change, the burst's tail latency falls well below the baseline, and you can say what the slowest callers still wait on.

D3 R3 , not recorded yet
After your change, run make burst again and compare its store_reads and p99_ms with D2.

expectEvery caller still gets the price, store_reads is far below the baseline and matches what your design allows, and p99_ms has fallen. Say what the slowest callers still wait on.

done whenThe design choice and what it does when a refresh fails are written down next to the code.

P4 · Keep checkout usable without Redis

What the stakeholder promised decides what an outage looks like to a shopper.

R4
hidden until you ask the stakeholder
ask the stakeholder
start the project to ask

D4 R4 , not recorded yet
Request sku-1 so it is cached. make redis-down, request it; make store-down, request it again within 30 seconds of its last read from Postgres. Then a SKU you never requested, and sku-1 once more 30 seconds later. make redis-up and make store-up bring them back.

expectThe first two requests answer in under a second with a price; the second is the last known one, marked stale. The never-requested SKU, and sku-1 30 seconds later, get your documented missing-price answer.

done whenThe outage controls and the freshness policy are written down for whoever is on call next.

Questions for the stakeholder

They reveal no requirement, but the answers shape the design.

start the project to ask

Technical guidance

▸Measure cold and warm requests separately; an average of the two hides the slow ones.
▸Keep the burst on one hot key, so the store-read count measures coordination.
▸Write down how stale a price a shopper can live with before you choose between waiting, serving stale, and refreshing early.

Done when

Every drill above passes, and:

The design notes say what your final choice costs in freshness, in failure behavior and under load.

Solutions

No solutions published yet. Ship the project and yours will be the first.