The API nobody could see inside

Friday evening

Lumen Home sells home goods. Its catalog API handles about 2,400 requests a minute at the evening peak, and checkout reads current prices from it, so when the catalog fails, carts stall even though checkout itself is fine.

Last Friday the price store behind the catalog started failing, and 14% of catalog requests failed for eleven minutes. The team heard about it from customer tickets. The service had one application log of free-form messages and no metrics, and nobody could tell whether requests were failing, slowing down or piling up. By the time they found the fault, 9% of add-to-cart attempts had failed.

What you're changing

Make the service expose its four golden signals, put them on one dashboard, and write the alert rules that page the on-call before the customers notice. The catalog's responses stay as they are.

What the starter gives you

The catalog service (Python and FastAPI) and, around it, everything else already wired: a fake price store with fault controls, background traffic, Prometheus, Alertmanager, a stand-in pager and Grafana. Prometheus scrapes GET /metrics on the service and loads every rule file in monitoring/rules/; Grafana loads every dashboard in monitoring/dashboards/. Both folders are empty.

make up starts it all. make fault makes every fifth price-store call fail and every call take 250 ms, make fault-latency makes every call take 2.5 seconds, and make clear-fault turns both off. make load sends 100 requests and prints one summary line. make rehearse and make rehearse-latency turn a fault on while the background traffic runs and print how many seconds passed before the pager got a page. make contract checks the catalog still answers as it did.

The rules Prometheus loads are the ones the rehearsals run against, and a rehearsal can't wait an hour: scale the windows down for the lab, and write the production windows beside them.

You can port the service to another language: keep its routes, port 8000 inside the container and STORE_URL, and run make contract until it passes.

Tracing, log shipping and running the monitoring stack with high availability are out of scope; they are questions for your write-up. For the deploy, the service with its /metrics reachable is enough.

The work, in 2 phases

Each phase says what must be true and the drill that proves it. You close a phase when its drills pass.

P1 · See the service's shape

The four golden signals, labelled so you can read them by route, on one dashboard.

R1
The service exposes request rate, error rate, latency and saturation, with labels that tell the catalog route from the rest and no label whose values grow with the traffic.
R3
The catalog answers exactly as before: `make contract` passes after your change.

D1 R1 , not recorded yet
Run make load three times: with no fault, after make fault and after make fault-latency. Then make clear-fault. In Prometheus at localhost:9090, find the series for traffic, errors, latency and saturation on the catalog route.

expectThe four signals tell request volume, failed responses, the response-time distribution and resource pressure apart, by route, and no label holds a SKU or a request id.
D2 R1 , not recorded yet
Open your dashboard at localhost:3000 during make fault-latency and count the panels that answer the four golden-signal questions for the catalog route.

expectAt least four panels, one each for traffic, errors, latency and saturation on the catalog route.
D3 R3 , not recorded yet
Run make contract with your instrumentation in place.

expectAll four contract tests pass.

P2 · Page before customers do

An SLO and burn-rate alerts that page in both rehearsals, fast enough to beat the tickets.

R2
hidden until you ask the stakeholder
ask the stakeholder
start the project to ask

D4 R2 , not recorded yet
Run make rehearse, then make rehearse-latency, and read the first_page= line each one prints.

expectBoth rehearsals page, and the alerts that page are declared in monitoring/rules/ against one catalog objective, with a fast and a slow burn window.
D5 R2 , not recorded yet
From the same two rehearsals, write down the seconds= on each first_page= line and the annotations printed under it.

expectEach page arrives within 60 seconds of the fault, and its annotations point the on-call to the evidence to open first.

Questions for the stakeholder

They reveal no requirement, but the answers shape the design.

start the project to ask

Technical guidance

▸Pick metric labels that answer a question, and keep SKUs and request ids out of them.

▸Write down the SLO's good event and its windows before you tune a threshold.

▸Run both rehearsals, the failing calls and the slow calls.

▸`make reload` rereads your rule files; Prometheus's Alerts page shows what they evaluate to.

Done when

Every drill above passes, and:

The rules and the dashboard are in the repository and load from a clean clone with `make up`.

A `RUNBOOK.md` says who gets paged, what to look at first and how to clear the rehearsal fault.

# The API nobody could see inside