Train like the backend engineer you're becoming.

Real backend projects — not tutorials. Pick a spec, build it on your own machine, then break it on purpose and write down what happened. The failures are the training.

01 pick a project 02 build on your machine 03 break it, then document

Start with GitHub or read the projects below first.

By signing in you agree to the privacy policy.

# pay-once · drill D1

$ make up

✓ the system comes up as it runs in production

✓ the load generator is ready

# then, on purpose:

Send the same charge body twice with one idempotency key and inspect both responses and the charge list.

D1 Both responses describe the same charge, and the second request does not create a new charge.
D2 The storm produces one charge for the key and reports zero duplicate charges.
D3 The second request returns a conflict and the original charge remains the only charge for that key.

record what happened, then write it up

Projects

Real systems. One project at a time.

Real backend projects — not tutorials. Pick a spec, build it on your own machine, then break it on purpose and write down what happened. The failures are the training.

ALL · 9 WARMUP · 2 INTERMEDIATE · 4 ADVANCED · 3 INCIDENTS · 3

01 · project Warm-up

The payment that happened twice

A mobile timeout made one customer pay twice. Make retries safe without hiding a changed request behind the same key.

Python FastAPI

~4h Start →

02 · incident Warm-up

The reports that slowed Tuesday

A new report filter turned Tuesday's p99 into a customer wait. Find the evidence, fix the query path, and prove the number comes back.

Python PostgreSQL

~3h Start →

03 · project Intermediate

The quota that multiplied

Three API instances quietly gave every client three quotas. Replace the local counter with one shared window and keep clients informed when to retry.

Python Redis Docker

~10h Start →

04 · project Intermediate

The orders that never reached the warehouse

BrasaCart accepted orders that vanished between checkout and the warehouse; make the handoff survive a broker outage and a repeated delivery.

Python RabbitMQ sqlite Docker

~12h Start →

05 · incident Intermediate

Twice the shipping email

A worker restart made RotaSul send the same shipping notice twice; trace the delivery, fix the invariant, and prove the count is zero.

Python RabbitMQ Docker

~10h Start →

06 · project Intermediate

The API nobody could see inside

LumeCasa discovered a catalog outage through customer tickets; make the service explain itself and page before the next incident spreads.

Python FastAPI Docker Prometheus Grafana

~14h Start →

07 · project Advanced

The checkout prices that vanished after a deploy

A deploy emptied Mira's price cache and a burst of shoppers overwhelmed the store; make cold reads safe, measurable, and useful during a Redis outage.

Python FastAPI Redis PostgreSQL

~24h Start →

08 · project Advanced

The notification queue that made OTPs wait

Bellbird's newsletter backlog made one-time codes wait while providers failed and throttled; separate urgency from delivery risk and prove the result.

Python FastAPI Redis

~28h Start →

09 · incident Advanced

The wallet that spent the same credit twice

Hearth's wallet allowed one customer to spend the same credit twice; reproduce the negative balance, rule out the slow PSP, and restore the invariant.

Python FastAPI PostgreSQL

~22h Start →