Concurrency vs Throughput

Concurrency is the number of operations in progress, while throughput is the number completed per unit time. They are related through time spent in the system, but higher concurrency does not guarantee higher throughput.

Concurrency describes how many operations are being worked on at the same time. In a backend service, this might mean the number of requests currently in progress. It is a count, not a rate.

Throughput describes how many operations complete per unit of time, often measured in requests per second (RPS). It is a rate. Offered load—the rate at which clients send requests—is distinct from throughput, which counts completed requests.

The two are related by Little's Law in a stable system: average in-flight work equals arrival rate multiplied by average time in the system. If a service completes 50 requests per second and each request takes 0.4 seconds on average, it has about 20 requests in flight on average. This relationship assumes a sufficiently steady system and consistent measurement boundaries.

Load tests should track both metrics alongside latency and errors. Increasing concurrency can raise throughput when the service has spare capacity, but beyond a point it may mostly increase waiting, latency, timeouts, or failures. Interpret results together rather than treating a larger concurrency setting as proof of higher capacity.

Key points

• Concurrency is the amount of work in progress; throughput is completed work per unit time.
• Measure throughput as successful completions, not merely requests sent.
• Little's Law relates average in-flight work to arrival rate and average time in the system.
• More concurrency can increase throughput until a bottleneck is reached; afterward it may mainly increase latency and failures.
• Read concurrency and throughput alongside latency, errors, and offered load.

Interview questions

What is the difference between concurrency and throughput?

Concurrency is how many requests are in progress at once, while throughput is how many requests finish per unit of time. For example, a service might have 100 requests in flight but complete only 20 requests per second. I would also check latency and errors, because high in-flight work can indicate queuing rather than useful capacity.

How does Little's Law connect concurrency, throughput, and latency?

Little's Law states that average work in the system equals the arrival rate multiplied by the average time spent in the system. For a steady workload, 50 requests per second with an average system time of 0.4 seconds implies about 20 requests in flight on average. The rate and time must refer to the same system boundary, and their units must be compatible.

A load test shows increasing concurrency but flat throughput. How would you investigate?

I would first check whether the test is actually driving the intended offered load and whether the service is completing requests successfully. Then I would compare throughput, latency percentiles, in-flight requests, and error rates as load increases. If concurrency keeps rising while throughput flattens and latency worsens, that suggests queuing or a bottleneck rather than useful throughput growth. I would inspect resource and dependency metrics to locate the constraint.

How can a load-test model affect the relationship you observe between concurrency and throughput?

A fixed-concurrency test keeps a target number of operations active, so its request rate can change as response times change. A fixed-arrival-rate test attempts to send work at a specified rate, making it useful for checking performance under a defined offered load. Under saturation, a closed-loop fixed-concurrency client may send fewer requests because it waits for responses before starting replacements. I would choose the model that matches the production traffic question and watch for dropped or delayed test traffic.