Distributed Rate Limiting

Distributed rate limiting coordinates request quotas across multiple application instances. It prevents clients from exceeding a shared limit even when requests are handled by different servers.

In a distributed application, each application instance sees only part of the traffic. A local counter therefore cannot enforce a global quota reliably: a client could send requests to several instances and exceed the intended limit. Distributed rate limiting introduces shared coordination, commonly through a fast centralized store such as Redis.

A limiter typically identifies a client with a key such as `userId`, API key, or IP address and associates that key with request state. For each request, the system checks whether the client still has capacity and either allows the request or rejects it, commonly with HTTP `429 Too Many Requests`.

Common algorithms include fixed window, sliding window, token bucket, and leaky bucket. The choice affects burst behavior, accuracy, memory usage, and implementation complexity. Token bucket is often useful when controlled bursts should be permitted while maintaining a long-term rate.

Distributed enforcement introduces consistency and failure-mode concerns. Atomic operations are important when multiple application instances update the same limiter state concurrently; otherwise, simultaneous requests can overshoot the quota. The design must also decide what happens if the shared limiter is unavailable, balancing availability against protection from excessive traffic.

Key points

• Use shared state when the quota must apply across instances.
• Atomic updates prevent concurrent requests from bypassing the limit.
• Choose the algorithm based on burst, precision, and resource requirements.
• Return `429 Too Many Requests` when a request is rejected by the quota.
• Define fail-open or fail-closed behavior for limiter-store failures.

Interview questions

Why can't a local in-memory rate limiter reliably enforce a global limit in a multi-instance service?

Each instance maintains its own independent counter. A client can distribute requests across several instances and remain below each local threshold while exceeding the intended global quota. Shared state or a coordinated enforcement layer is therefore needed for a global limit.

How would you implement distributed rate limiting with Redis?

I would use a deterministic key such as `rate:{clientId}` and store the state needed by the selected algorithm in Redis. The request decision and state update should be performed atomically, using an appropriate Redis command sequence or Lua script when multiple operations must be combined. Keys should have expiration aligned with the algorithm's time semantics. The service should return `429` when the limit is exceeded.

Which rate-limiting algorithm would you choose for an API that permits short bursts but needs a sustained average rate?

A token bucket is a natural fit because tokens accumulate at a configured rate up to a maximum capacity. Requests consume tokens, allowing a bounded burst when tokens have accumulated. The refill rate controls the sustained throughput while bucket capacity controls the burst size. The exact parameters should be derived from the API's workload and downstream capacity.

What consistency problems can occur in distributed rate limiting?

Multiple instances can concurrently observe available capacity and then update the same limiter state, causing the effective quota to be exceeded. Network delays, retries, replication lag, and partial failures can also make state temporarily inconsistent. Atomic operations reduce race conditions, while the architecture must explicitly define acceptable overshoot and failure behavior. For strict limits, the coordination mechanism and algorithm must be selected with their consistency guarantees in mind.

How would you design rate limiting for multiple dimensions, such as per-user and per-IP limits?

I would model each dimension as a separate limit and evaluate all applicable policies for a request. For example, a request might need both a user-level allowance and an IP-level allowance. The implementation should avoid partial updates that consume one quota while failing another, especially when the policies are intended to be enforced together. I would also define precedence, response metadata, and observability so operators can determine which policy caused rejection.