Burst Capacity in Rate Limiting

Burst capacity is the amount of short-term traffic a rate limiter can allow beyond its sustained rate. In a token-bucket limiter, it is usually represented by the bucket's maximum token count.

A sustained rate describes how quickly requests can be admitted over time. Burst capacity determines how much traffic can be admitted in a short interval, often because unused capacity has accumulated or the limiter starts with available allowance.

In a token bucket, tokens refill at a configured rate up to a maximum capacity. Each request consumes one or more tokens; a request is admitted only if enough tokens are available. The refill rate sets the long-term pace, while the capacity sets the largest available short-term allowance.

For unit-cost requests, a bucket with refill rate r and capacity B can admit at most roughly rT + B requests over an interval of length T, assuming it begins full. A burst can therefore use capacity quickly, but it does not create a permanently higher sustained rate.

Burst capacity is useful when legitimate traffic arrives unevenly, such as when clients send a small batch of requests together. A large burst allowance can also overload a backend, so it should be chosen alongside service capacity, request cost, and the behavior of clients and retries.

Key points

• The sustained rate controls replenishment; burst capacity controls the maximum accumulated allowance.
• A token bucket admits a request only when enough tokens are available.
• A larger burst capacity tolerates bigger short-lived spikes but may increase instantaneous load.
• Burst capacity is not the same as a concurrency limit or a guarantee that the service can handle the burst.
• The exact behavior depends on the limiter algorithm; the term is not implemented identically by every rate limiter.

Interview questions

What does burst capacity mean in a rate limiter?

It is the short-term allowance for requests above the steady replenishment pace. In a token bucket, it corresponds to the maximum number of tokens the bucket can hold. Requests can consume those tokens quickly, but new allowance returns only at the configured refill rate. It should reflect both expected traffic patterns and what the service can handle.

How does a token bucket enforce a rate and a burst limit?

The bucket stores up to a configured number of tokens and refills them at a configured rate. An incoming request consumes its cost in tokens if enough are available; otherwise, the limiter rejects or delays it. The refill rate controls long-run admission, and the bucket capacity bounds the accumulated allowance. If request costs vary, the limiter can charge different token amounts for different operations.

How would you implement a burst-capable limiter across multiple Python service instances?

I would store the bucket state in a shared system or use a gateway that owns the decision, rather than giving each process an independent full bucket. Updating token count and timestamp must be atomic; otherwise, concurrent requests can spend the same tokens. I would use a monotonic time source where the state is maintained and define how state behaves during store failures or key expiration. I would also measure rejection rates and backend load, and test behavior under concurrent traffic.

How would you tell whether a burst setting is too permissive?

I would compare admitted traffic peaks with the service's measured capacity and latency objectives. I would look at rejection rates, queue depth, error rates, and latency during bursts, not just average request rate. A configuration that avoids rejections but causes overload is not successful. I would tune the burst and sustained rate separately, then validate the change with realistic load tests.