backendgym backendgym — projects Sign in with GitHubSign in

projects /06-four-signals-and-a-pager

# The API nobody could see inside

LumeCasa discovered a catalog outage through customer tickets; make the service explain itself and page before the next incident spreads.

Intermediate ~14h PythonFastAPIDockerPrometheusGrafana phase 1 of 3 · Follow one failed request
Sign in with GitHub
four-signals-and-a-pager / spec.md v1.0

The incident

LumeCasa is a home-goods retailer whose catalog API serves about 2,400 requests a minute during the evening shopping peak. Checkout depends on it for current prices, so a catalog outage becomes a stalled cart even when the checkout process itself is healthy.

Last Friday a dependency fault made 14% of catalog requests fail for eleven minutes. The team found out from customer tickets, not from the service. The only application log had free-form messages, there was no request correlation, and no one could tell whether the time was spent in the API, the dependency, or a saturated process. Nine percent of add-to-cart attempts failed before the team found the fault.

Instrument the API, give the team a dashboard that answers the four golden-signal questions, and make an alert fire during a rehearsed fault. Keep the catalog response contract and business logic unchanged.

Lab boundary

The starter supplies a containerized FastAPI service and a load generator. The fault controls simulate dependency errors, latency, and CPU pressure. OpenTelemetry instrumentation, Prometheus, Grafana, Tempo, SLO policy, alert rules, dashboard assembly, and deployment are graded work; production high availability and long-term storage are design-doc material.

Starter

Use the template at https://github.com/Tserewara/starter-four-signals-and-a-pager.

P1 · Follow one failed request

Give the team a reliable path from a customer report to the failing request.

  1. R1 A request can be followed through a structured log and a trace using one correlation id, including a request that ends in a fault.
Failure drills
  • D1Inject a catalog error and follow one failed request from the load generator into the service logs and trace backend.
  • D2Send 100 requests with the fault enabled and calculate the percentage whose logs and traces contain the same correlation id.
  • ○A failed request has a stable correlation path through the service evidence.

P2 · See the service shape

Turn traffic, failures, latency, and pressure into an operator view.

  1. R2 The service exposes request rate, error rate, latency, and saturation signals with labels that support route-level diagnosis.
Failure drills
  • D3Run the load generator once with normal traffic and once with an error and latency fault; inspect the metric store for route-level traffic, errors, latency, and saturation series.
  • D4Count the dashboard panels or signal queries that answer the four golden-signal questions for the catalog route.
  • ○The dashboard and metric queries answer the four golden-signal questions for the catalog route.

P3 · Page before customers do

Tie the signals to an SLO and rehearse the response path.

  1. R3 hidden until you ask
  2. R4 The monitoring stack and instrumented API can be deployed from the repository without changing the catalog response contract.
Failure drills
  • D5Inspect the SLO and alert-rule configuration, then use the load generator to create the supplied error and latency faults over the configured windows.
  • D6Inject the catalog fault, start the documented load, and measure the time from fault injection to the first alert notification while the dashboard shows the failing signal.
  • ○The deployed monitoring path preserves the API contract and leaves an operator with a tested alert route.

Technical guidance

  • ▸Start with one failed request and follow its id before building a wall of dashboards.
  • ▸Choose metric labels that answer a question without turning every request id into a new time series.
  • ▸Write down the SLO event and window before tuning alert thresholds.
  • ▸Rehearse both an error fault and a latency fault; one signal can look healthy while another is failing.

Definition of done

All of them, or it isn't shipped.

  • The failed request has one correlation id in structured logs and a matching trace with the fault visible as a timed span. — not recorded yet D1
  • correlated requests over 100 percent at the same load — not recorded yet D2
  • The four signals distinguish request volume, failed responses, response-time distribution, and resource pressure without using an unbounded request-id label. — not recorded yet D3
  • golden signal views over 4 panels at the same load — not recorded yet D4
  • The SLO, fast-burn alert, and slow-burn alert are declared in the repository and their expressions point to the same catalog objective. — not recorded yet D5
  • fault-to-alert time under 60 seconds at the same load — not recorded yet D6
  • The repository declares the API, telemetry collectors, metric store, dashboard, alert rules, and deployment configuration as one reproducible stack. — not recorded yet
  • The catalog endpoint returns the same status codes and JSON fields after instrumentation. — not recorded yet
  • The runbook says who receives the page, what evidence to inspect first, and how to clear the rehearsal fault. — not recorded yet

nothing recorded yet — running the drills fills these in

spec v1.0 · drafted with AI, human-reviewed

Solutions

No solutions published yet. Complete the project and yours opens the thread.