Concurrency Limiting

Concurrency limiting caps how many operations may be active simultaneously, protecting constrained resources such as databases or downstream APIs. It creates backpressure only when paired with a deliberate policy for work that cannot acquire a permit immediately.

Concurrency limiting places a ceiling on the number of operations simultaneously using a resource, such as a database connection pool, outbound HTTP client, or CPU-heavy worker. It is a form of backpressure: when all permits are occupied, additional work waits, is rejected, or is shed according to the system’s policy.

In Python’s `asyncio`, an `asyncio.Semaphore(n)` is a common way to guard a region of asynchronous work. `async with semaphore` acquires a permit and releases it even if the block exits with an exception. A semaphore is generally preferable to manually tracking an active count because acquisition and release are coordinated safely.

The limit should reflect the constrained resource and the service’s latency and throughput goals. A value that is too high can overload a downstream dependency; one that is too low can leave capacity unused. Measure queue wait time, in-flight work, errors, and downstream latency, and account for limits across all application instances rather than tuning only one process.

A concurrency cap alone does not bound total demand. If the application creates an unbounded number of tasks, many can accumulate while waiting for permits, consuming memory and increasing latency. Pair the cap with bounded queues, request deadlines, cancellation, or rejection when the system must also control waiting work.

Key points

• Cap simultaneously active work to protect scarce resources.
• Use `asyncio.Semaphore` to guard asynchronous operations in Python.
• A cap does not bound queued or waiting work; combine it with an overload policy.
• Tune for aggregate downstream capacity, not just one process.

Interview questions

What does concurrency limiting do, and how is it different from rate limiting?

A concurrency limit caps how many operations are active at once, while a rate limit caps how many may start over a time window. I would use a semaphore around calls that consume a scarce resource, such as database or HTTP connections. When permits are exhausted, callers wait or follow an explicit overload policy. I would also bound the waiting queue if demand can grow without limit.

How would you implement an asynchronous concurrency cap in Python?

I would use `asyncio.Semaphore` for a fixed number of concurrent asynchronous operations and acquire it with `async with` around the resource-consuming section. This ensures the permit is released on normal exit or exceptions. I would avoid holding the permit across unrelated work, because that reduces useful capacity. The semaphore limits active entrants, not the number of tasks created upstream.

How would you select and operate a concurrency limit in a distributed backend?

I would first identify the actual bottleneck and measure downstream latency, errors, saturation, and queue wait time. Then I would tune the cap against the downstream service’s capacity and the application’s latency objectives, testing under representative load. The cap should be considered at fleet level: a per-process value multiplied by the number of instances may exceed the dependency’s safe capacity. I would also define deadlines, cancellation, and overload behavior so queued requests do not wait indefinitely.

Does a semaphore alone provide complete backpressure?

A semaphore limits the number of coroutines inside a critical region, but it does not stop code from creating a large number of tasks that then wait for permits. That can still consume memory and inflate latency. A bounded queue with a fixed number of workers limits both queued items and active consumers, and lets producers experience backpressure. Alternatively, the service can reject excess requests or apply deadlines.