API Observability with Prometheus

The situation

You have an HTTP API that works. Nobody can tell you whether it is healthy right now, and when it degrades you find out from a user.

Instrument it so that a single dashboard answers three questions during an incident: what is failing, how badly, and since when.

The service you instrument is yours — an API you have already built, or a small one you write for this. Choosing it is not the work.

The instrumentation library is five lines. Everything below is why this is still a real problem.

Cardinality. A label whose value comes from the request — a path containing an id, a user handle, a query parameter — multiplies your series by every distinct value ever seen. Prometheus does not forget them quickly.
Buckets. Default histogram buckets are somebody else's latency distribution. If every request lands in one bucket, your p99 is a fiction with a number attached to it.
Averages. A mean over a bimodal distribution describes no request that ever happened.
Workers. More than one process behind one port means more than one registry, and a scrape reaches whichever process answers.
Counter resets. A counter returning to zero on deploy is normal. A panel that reads it as a level instead of a rate is not.

A dashboard is worth what it tells you under stress, so find that out before you call this done:

Make one endpoint slow deliberately, and watch which panel notices first — and how long it takes.
Send the traffic that would explode cardinality if you had labelled naively, then check your series count.
Restart the service mid-load and see what your rate panels do at the seam.
Stop the scrape target and learn how your dashboard shows no data versus zero.

Requirements

What must be true — the how is yours.

R1 Every HTTP request is counted by method, route template and status class, and its duration is recorded as a histogram.

R2 Prometheus scrapes the service on a schedule, and the series survive a restart of the app. rehearsed by D3 · D6

R3 One Grafana dashboard shows request rate, error rate and latency percentiles, and is readable by someone who did not build it. rehearsed by D1

R4 Your p99 is defensible: you can say why your histogram buckets fit the traffic you actually serve. rehearsed by D5

R5 Series count stays bounded no matter what a client sends — a careless or hostile caller cannot grow your cardinality. rehearsed by D2

R6 At least one alert rule fires on a condition you have reproduced yourself. rehearsed by D4

Technical guidance

▸Measure the symptom before the cause: rate, errors and duration answer more incidents than any custom gauge.

▸Every label multiplies your series. Decide what a label may contain before you add it, not after Prometheus grows.

▸Bucket boundaries are a design decision — pick them from latency you have measured, not from the library default.

▸Instrument at one boundary you control, so a request cannot be counted twice or missed entirely.

▸Decide what a restart means for each metric before you build a panel on top of it.

Definition of done

All of them, or it isn't shipped.

A latency panel moves, and you can say how long it took to become visible. D1

Series count stays flat while request count climbs. D2

The rate panels stay readable across the restart and no panel reports a negative or absurd value. D3

The alert fires, and you know how long it took. D4

Requests are spread across several buckets, so a p99 read from them means something. D5

The dashboard distinguishes 'no data' from 'zero', and you can tell which you are looking at. D6

One command brings up the API, Prometheus and Grafana together.

The dashboard lives in the repository as code, not only in Grafana's database.

nothing recorded yet — running the drills fills these in