The Limiter That Caused Our Outage

Community Article
Community articles are authored by SitePoint Premium contributors. Content is screened before publication, and SitePoint reserves the right to moderate or remove articles that violate our guidelines. Views expressed are those of the authors and do not necessarily reflect those of SitePoint.

Python Rate Limiting: The Limiter That Caused Our Outage

SCShoumik ChakravartyPublished inCloud·APIs·Python·Scaling·
September 4, 2026
·Updated:September 4, 2026

The AI briefing for Developers

Stay up to date with AI tools, model releases, and developer workflows that matter.

Weekly. Free. One click to leave.

SitePoint Premium
Stay Relevant and Grow Your Career in Tech

• Premium Results
• Publish articles on SitePoint
• Daily curated jobs
• Learning Paths
• Discounts to dev tools

7 Day Free Trial. Cancel Anytime.

I’ve seen a rate limit look fine on paper and then fall apart the moment prime-time traffic hit. In video streaming, milliseconds matter – a user clicks Play during a live event and expects the stream to start instantly. We once shipped a bad client-side retry loop that never backed off. Every error made the client hammer the API harder, turning millions of boxes into a self-inflicted DDoS. Server resources started burning, alarms lit up across the fleet, and for a while we genuinely thought someone was attacking us. Only after digging into the call patterns did we realize the “attack” was our own code.

It turned out the bad client was the real culprit – but the part that fooled us was the firewall’s rate-limit policy. We thought it would shield the service, and instead it made the whole thing look worse. Every rejection pushed each client box to retry immediately, so the calls kept multiplying. What was supposed to be protection turned into an illusion, and the “defense” amplified the very surge the broken client was creating.

Most rate limiting tutorials show you how to reject requests. Almost none show you what rejecting does to the client on the other end. So let’s build one badly, then fix it three times.

Attempt 1: Fixed-window counter

Here’s the limiter almost everyone reaches for. Count requests per client, reset the count every minute, reject anything over the quota.

from fastapi import FastAPI, Request, HTTPExceptionimport math, timeapp = FastAPI()QUOTA, WINDOW =100,60counters ={}@app.middleware("http")asyncdefrate_limit(request: Request, call_next):    key = request.client.host    now = time.time()    start = math.floor(now / WINDOW)* WINDOW    w_start, count = counters.get(key,(start,0))    if w_start != start:        w_start, count = start,0    if count >= QUOTA:        raise HTTPException(429,"Too Many Requests")    counters[key]=(w_start, count +1)    returnawait call_next(request)

Ten lines, obvious, and wrong in two ways.

The first is the one people know about. The window resets on a wall clock, so a client can spend its whole quota at 11:59:59 and spend it again at 12:00:00:

allowed at 59.9s: 100
allowed at 60.1s: 100
-> 200 requests in ~0.2s against a “100 per 60s” limit

Your limit says 100 a minute. Your API is serving 200.

The second failure is the one that cost us a night. Every throttled client becomes eligible again at exactly the same instant, because they all share the same clock boundary. A reset-based limiter doesn’t merely permit a thundering herd. It schedules one, and it puts the invitation in the response.

Where does the limiter actually live?

Before fixing the algorithm, it’s worth asking where it runs, because that outage happened at a firewall and the code above runs inside a Python process.

The reflex is that pushing rate limiting out to the edge makes it someone else’s problem. Our outage is the counter-example. The edge had the limiter, and the edge caused the amplification.

This is the part worth carrying through the rest of the article: the algorithm’s behavior is the same wherever you enforce it. A fixed-window reset synchronizes clients whether it’s a firewall policy, a gateway rule, or ten lines of middleware. Rejecting cheaply at the edge doesn’t make a reset boundary safe. It just makes the herd arrive faster.

Most real systems run limits at more than one layer anyway: a coarse abuse limit at the edge, a per-customer quota further in.

That outage forced us to rethink where the limiter actually lives. We kept a coarse abuse limit at the edge, but we stopped pretending it was the thing keeping us safe. The real protection moved inward, closer to the application, where we actually have context – who the customer is, what plan they’re on, and how expensive the call is. The edge still rejects the obvious garbage, but the meaningful limits now run in middleware.

Attempt 2: The token bucket

Give every client a bucket that refills at a steady rate. Each request costs a token. No tokens, no service.

capacity 10, refill 1/sec

immediate burst allowed: 10
allowed 5s later:         5

Two things improve immediately. Clients can burst up to capacity, which is usually what you actually want – real traffic is lumpy, and a limiter that punishes every burst is a limiter people route around. And the refill is continuous rather than a cliff at a shared boundary, so clients recover at slightly different moments instead of all at once.

That second property is the fix for the failure in the opening. Nobody is handed the same reset instant.

The cost is state: a token count and a timestamp per client.

Attempt 3: GCRA, one timestamp per client

Almost every rate limiting article stops at the token bucket. There’s a better default, and it comes from 1990s telecom networking.

GCRA – the Generic Cell Rate Algorithm – asks a different question. Instead of “how many tokens are left?”, it asks “what’s the earliest time this client may send its next request?” It stores that single timestamp and nothing else.

Each allowed request pushes the timestamp forward by the average spacing your limit implies. Ten requests per ten seconds means one second per request. If the clock has passed the stored time, allow it. If not, reject – and the gap tells you exactly how long the client should wait.

10 requests per 10s

instant burst allowed:      9
sustained 1/s for 20s:      20/20 allowed
hammering at 10x for 20s:   29/200 allowed

A quota of 10 allowing 9 in an instant looks like an off-by-one, but it isn’t. The burst a client gets is one window’s worth of credit minus the spacing that the request being served costs. Ten seconds of credit, one second per request, nine requests through the door before the tenth has to wait. Widen the clamp if you want a fuller burst; that parameter is the burst tolerance dial.

GCRA per client: 1735689600.0 (one float)

TokenBucket per client: (4.0, 1735689605.0) (tokens + timestamp)

Go back to that third line of output, the one where a client hammers at ten times the limit. It gets 29 requests through, the same as the token bucket would allow. The difference is what happens to it: the effective window shrinks while it misbehaves, rather than resetting on a clock. Badly behaved clients get smoothed out instead of queued up for a synchronized release.

There’s no refill task either. The clock moving forward is the refill.

We ended up enforcing at two layers, and that outage made the choice clearer. We kept a coarse abuse limit at the edge because it’s cheap and it filters the obvious garbage, but anything that actually matters moved inward. The gateway was pushing 500,000 requests a minute, and per-client state at that layer wasn’t free: every Redis round-trip, every lookup, every hop showed up in latency during prime-time traffic. Inside the application we finally had the context we needed, but we also had to keep the memory footprint tiny. That’s why we stayed with a token bucket instead of GCRA. One float per client is elegant, but operational simplicity mattered more: predictable behavior, no surprises, and tooling everyone already understood. And the real fix wasn’t the limiter anyway – it was stopping the clients from retrying themselves into a stampede.

The bit that breaks the moment you deploy

All three limiters above store state in a dictionary in one process.

Run uvicorn –workers 4 and your “100 requests per minute” limit becomes up to 400 a minute, because each worker has its own dictionary and none of them talk. Scale to eight pods and it’s up to 800. How close you get to the ceiling depends on how a client’s requests are spread across workers – one busy client hitting all four workers evenly gets the full multiple; a quiet one may never notice. Behind a load balancer across pods, assume the full multiple.

Either way, the limit in your config is not the limit your API enforces, and nothing warns you.

The fix is shared state, done atomically. This is wrong:

count = redis.get(key)
redis.set(key, int(count) + 1)

Two requests can read the same value before either writes, and you undercount under exactly the load you care about. This is right:

import math, timenow = time.time()bucket = math.floor(now / WINDOW)* WINDOW     key =f"ratelimit:{client_id}:{bucket}"pipe = redis.pipeline()                        pipe.incr(key)                                 pipe.expire(key, WINDOW, nx=True)count, _ = pipe.execute()if count > QUOTA:    raise HTTPException(429,"Too Many Requests")

Three things worth knowing about those eight lines.

The window has to be in the key. Without the bucket suffix, EXPIRE NX starts the clock on each client’s first request, which quietly turns this into a per-client rolling window rather than the wall-clock one we built earlier. That happens to dodge the synchronized boundary, which is a fine design, but it’s a different limiter, and you should choose it deliberately rather than inherit it from a TTL.

EXPIRE … NX needs Redis 7.0 or newer. The NX/XX/GT/LT options arrived in 7.0. On 6.x this errors, and you want the older idiom instead: set the TTL only when INCR returns 1.

“Pipeline” doesn’t mean atomic by itself. redis-py wraps pipelines in MULTI/EXEC by default, which is what makes this safe. Construct it with pipeline(transaction=False) and you get batching without atomicity – faster, and wrong for this.

Which raises the question nobody enjoys answering: what happens when Redis is down?

Run that code with Redis unreachable and you’ll find out. pipe.execute() raises ConnectionError, it propagates straight out of the middleware, and every request to your API becomes a 500. Not throttled, broken. That’s worse than either option, because you didn’t choose it.

The choice has to be explicit:

try:pipe = redis.pipeline()    pipe.incr(key)    pipe.expire(key, WINDOW, nx=True)    count, _ = pipe.execute()except redis.exceptions.ConnectionError:    returnawait call_next(request)        if count > QUOTA:    raise HTTPException(429,"Too Many Requests")

Fail open and you have no limiter during the incident where you need one most. Fail closed and your limiter has just taken down your API – which, given how this article started, is not a hypothetical. Whichever you pick, pick it deliberately and put it in a try block, because the default behavior is neither.

We chose to fail open, but only after tightening everything around it. The outage made the risk obvious: if Redis goes down and the limiter fails closed, you’ve just taken your own API offline during the moment you need it most. Failing open isn’t pretty, but it keeps the service alive, and the real protection comes from the layers around it – coarse limits at the edge, per-customer quotas inside, and client retry behavior that won’t turn a hiccup into a stampede. The limiter can wobble; the system around it can’t.

Tell the client how to behave

Here’s what we got wrong in the outage, reduced to one sentence: our limiter rejected requests without ever telling clients when to come back. So they came back immediately, forever.

A limiter that rejects without instruction is half-built.

Most tutorials will tell you to send RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset. That advice is out of date. The current IETF draft (revision 11, May 2026) replaced those three with two structured fields:

•   RateLimit-Policy – the policy itself, which should stay stable across responses: a policy name plus q (quota, required), w (window), qu (quota units), pk (partition key)

•   RateLimit – the current state for this response: policy name plus r (available quota, required), t (effective window), pk

Both are structured field lists, so a server can advertise several policies at once:

RateLimit-Policy: “permin”;q=50;w=60,”perhr”;q=1000;w=3600
RateLimit: “permin”;r=37;t=42

That says: fifty requests a minute and a thousand an hour; you have thirty-seven left on the per-minute policy, and forty-two seconds until that number changes.

If you built the GCRA limiter, you already have r and t. It produces them as a by-product of deciding whether to allow the request.

Then Retry-After on the 429. SitePoint has already covered the client side of this in Claude Code Rate Limits Explained: back off exponentially, add jitter, respect Retry-After. That advice is correct, and it only works if the server sends something worth respecting. A 429 with no retry hint gives a well-behaved client nothing to back off against, so it guesses, and every client guesses the same way.

Our clients weren’t badly written because they retried. They retried immediately because we never told them not to. Send a Retry-After, spread the values you send across a random interval, and the herd never forms. It’s a two-line change on the server, and it’s the one that would have prevented the outage this article opens with.

The failures you only see in production

The traffic graph from that night is still burned into my memory. Prime-time was already heavy, but the moment the bad client code rolled out, the curve bent upward in a way that didn’t look human anymore. It wasn’t a clean spike; it was a rising wall. Every client box that hit the retry loop added its own little staircase, and together they formed a pattern that looked exactly like an attack. The limiter at the edge became the bottleneck. It was rejecting fast enough to stay alive, but every rejection triggered another call, so the graph kept climbing even though the service wasn’t doing any real work.

The hardest part was telling legitimate traffic from the surge while it was happening. Live events always create bursts – people pause, rewind, jump back in, and you don’t want to mistake normal behavior for abuse. But the call pattern didn’t match anything we’d seen before. It was too uniform, too synchronized, too relentless. The limiter was doing its job, but the job itself had become the problem: the fixed-window reset lined up thousands of clients at the same boundary, and they all charged in together.

What to actually do

The application wasn’t overloaded; the protection layer was.

That’s the whole thing in six words, and it’s the part worth carrying out of here. A rate limiter is not just a gate, it’s a signal. Ours rejected traffic correctly and still made the outage worse, because it told a fleet of clients to come back at the same moment and said nothing about waiting.

Sensible defaults: token bucket or GCRA rather than a fixed window, shared state in Redis if you run more than one worker, the current RateLimit headers, and jitter on every retry hint you send.

And one thing to check today – whether your limit is quietly multiplied by your worker count, and whether anything you return tells a client when to come back.