The quota that multiplied

The incident

ParcelMint is a fictional shipping marketplace. Its public GET /v1/quotes API is used by 4,800 merchant integrations to price labels before checkout. Each API key is promised 10 requests per 10-second window.

During a carrier sale, one merchant sent a burst through the load balancer. We had three API instances, each enforcing the promise in its own memory, so the merchant received 30 admitted requests in one window. The carrier quotation service throttled us, label purchases failed for 8.4% of customers, and the carrier asked us to cut traffic until the afternoon.

Make the quota a property of the key, not of whichever instance received the request. Keep the public response useful when a caller is over limit. The work includes a deployable three-instance shape; the carrier itself stays simulated.

Lab boundary

The three API containers simulate separate deployed instances, and the local Redis container simulates the shared rate-limit store. The load generator distributes one key across all three instances. Managed Redis failover, multi-region clock policy, and quota administration remain design-work for the write-up.

P1 · Expose the multiplied quota

Run the public API as three instances and make the local limit's business failure visible.

R1 Across all three API instances, one API key can have at most 10 admitted requests in any 10-second sliding window.

Failure drills

D1Send 11 requests for one key to a single API instance inside 10 seconds and inspect the responses.
D2Run `make load` to distribute 30 requests for one key across all three instances in one 10-second window.
D3After the key reaches its limit, send one more request and inspect its status and Retry-After header.

○The broken three-instance behavior is recorded with a repeatable load command.

P2 · Make admission global

Choose the window and failure policy, then prove the shared decision under concurrency.

R2 hidden until you ask

R3 Concurrent requests cannot pass the shared admission decision in a way that exceeds the per-key limit.

Failure drills

D4Run a concurrent burst for one key through all three instance URLs, then inspect the implementation and its tests for the single admission decision.
D5Reset the stack, run the three-instance load again, and record admitted requests before the 10-second window closes.
D6Wait for the window to pass, then retry the same key and confirm the client can make progress without changing its key.

○The deployment and README describe the shared store and its failure policy.

Technical guidance

▸Start by routing one key across all three instances and count admissions in one window; the broken scope is the first fact to make visible.

▸Keep the window definition and the admission decision together. A shared counter without a window policy does not define the quota.

▸Treat Retry-After as part of the client contract, and decide what a Redis failure means before you test it.

▸The deploy shape is part of the lab: a local process is not evidence for a multi-instance guarantee.

Definition of done

All of them, or it isn't shipped.

The instance admits the first 10 requests and rejects the next request with HTTP 429. D1

admitted requests per key under 10 requests at the same load D2

The response is HTTP 429 and Retry-After is a positive number of seconds within the current window. D3

The shared decision is one atomic operation, so concurrency cannot create an extra admission between the check and the update. D4

admitted requests per key under 10 requests at the same load D5

A request is admitted after the window expires, and an over-limit response never tells the client to wait longer than the remaining window. D6

The three-instance deployment starts from a clean clone with one command and uses one shared rate-limit store.

The README documents the chosen sliding-window representation, atomicity boundary, and Redis failure trade-off.

The API preserves successful quote responses while adding the documented over-limit response.