The Dashboard That Stayed Green

The incident

Meridiano is a payments gateway. Merchants integrate once and call POST /v1/charges, and that one endpoint is the company's revenue. It deploys four times a day.

On 3 March a partner bank began refusing one range of card BINs. For eighty minutes about 4% of charges failed — roughly R$ 210 000 — and the dashboard stayed green the whole time. Average latency looked fine, because a declined charge fails fast. The error panel counted HTTP 500s, and the gateway was politely answering 200 with a declined body. Support found out from a merchant on Twitter.

Two days later somebody tried to close the blind spot in a hurry by adding merchant_id to every metric. Prometheus ran out of memory that night, and the team spent a morning getting their monitoring back before they could go near the original problem.

What you are building

Instrumentation and one dashboard an on-call engineer can use to say what is failing, how badly, and since when — and trust the answer.

Nobody wants a bigger dashboard. They want one that would have gone red on 3 March, and that stays standing under the traffic a merchant can point at it.

Starter

github.com/Tserewara/starter-dashboard-stayed-green — use it as a template, then make up.

The gateway is given, running four workers behind one port the way it does in production. So is a partner bank you can make slow, decline-happy or silent while traffic is running, and a load generator that can spread charges across as many merchants as you point it at.

There is no metrics code in it, no /metrics, no Prometheus, no Grafana, no dashboard and no alert rule. That part is yours, and how you build it is the thing you will be defending in the write-up.

Requirements

What must be true — the how is yours.

R1 A charge that failed the merchant is counted as a failure regardless of the HTTP status it was answered with. rehearsed by D1

R2 Latency is reported as a distribution, so a change affecting a minority of requests is visible without knowing in advance to look for it. rehearsed by D2

R3 No value a caller controls can create a new time series. Series count stays flat while request count climbs. rehearsed by D3

R4 Histogram buckets come from latency you measured on this traffic, and you can say why the p99 read from them means something. rehearsed by D4

R5 Counters resetting four times a day do not make a panel report a false level, a negative rate, or a hole in the graph. rehearsed by D5

R6 The dashboard and the alerting tell 'the service reported zero' apart from 'we received nothing', so monitoring does not go quiet exactly when the service disappears. rehearsed by D6

R7 At least one alert fires on a condition you reproduced deliberately, and you can show it would have fired on 3 March. rehearsed by D7

Technical guidance

▸Decide what the business calls a failure before you decide what to count. The transport's opinion and the domain's opinion disagree here, and that disagreement is the whole incident.

▸Every label multiplies your series by its distinct values, and Prometheus keeps them long after the traffic stops. Decide what a label may contain before you add it.

▸Bucket boundaries are a design decision. Pick them from latency you have measured on this traffic, not from the library default, which is somebody else's distribution.

▸Instrument at one boundary you control, so a request cannot be counted twice or missed entirely.

Definition of done

All of them, or it isn't shipped.

A panel moves and the on-call can see the failures without opening the logs. D1

A panel makes the slow minority visible; the number you would have quoted from an average does not. D2

Series count stays flat while request count climbs, and you can say what bounds it. D3

Requests are spread across several buckets rather than piling into one, so the p99 read from them describes real traffic. D4

The rate panels stay readable across the seam and no panel reports a negative or absurd value. D5

The dashboard shows absence distinctly from zero, and the alerting notices the silence instead of going quiet with it. D6

The alert fires, and you can state how long it took from the first failed charge. D7

One command brings up the API, Prometheus, Grafana and the load generator together.

The dashboard and the alert rules live in the repository as code, not only in Grafana's own database.

nothing recorded yet — running the drills fills these in