Latency-Throughput Trade-off

Latency measures how long an individual operation takes, while throughput measures how much work a system completes per unit of time. Increasing concurrency or batching can raise throughput, but it can also increase queueing, contention, and therefore request latency.

Latency is the elapsed time associated with processing a request or operation. In backend systems, it is often discussed using percentiles such as p50, p95, and p99 because averages can hide slow requests. Throughput describes completed work over time, such as requests per second or messages processed per second.

Concurrency can improve throughput when work is independent and resources are available. For example, a Java service can process multiple I/O-bound requests while other requests are waiting on a database or network operation. Beyond the useful concurrency level, additional work competes for CPU, memory, connections, locks, or downstream capacity, causing queueing and longer latency.

Batching combines multiple operations into one larger operation. This can reduce fixed per-operation overhead, such as network round trips or database calls, and therefore improve throughput. However, a batch may wait for more items to arrive, and a request near the beginning of the batch may wait for later items, increasing latency.

The relationship is not a universal rule that more throughput always means more latency. A system can sometimes improve both by reducing unnecessary work, increasing capacity, caching effectively, or removing contention. The important interview skill is identifying the bottleneck and explaining how concurrency, batching, queueing, and resource saturation affect both metrics.

In Java services, useful measurements include request latency percentiles, requests per second, CPU utilization, garbage-collection behavior, thread-pool utilization, connection-pool saturation, and downstream latency. Performance decisions should be based on measurements under a representative workload rather than on concurrency increases alone.

Key points

• Latency is the time taken by an individual operation; throughput is completed work per unit time.
• More concurrency can increase throughput until a resource becomes saturated, after which queueing and contention can dominate.
• Batching reduces per-item overhead but may add waiting time before a batch is processed.
• Use latency percentiles such as p95 and p99 instead of relying only on average latency.
• The useful concurrency level depends on the workload, available resources, and downstream limits.

Interview questions

What is the difference between latency and throughput?

Latency is the time required for an individual request or operation to complete. Throughput is the amount of work completed during a period, such as requests per second. They measure different aspects of performance and should not be treated as synonyms. A service can increase throughput while individual requests become slower because requests are waiting in queues.

Why can increasing concurrency improve throughput but hurt latency?

Additional concurrency can keep CPU, network, or other resources busy while some tasks are waiting, increasing useful work completed per unit time. Once a bottleneck becomes saturated, additional tasks cannot be served immediately and begin waiting. That queueing increases response latency, and contention can also reduce effective throughput. The practical goal is therefore to find a concurrency level that matches the system's bottlenecks rather than maximizing concurrency blindly.

How does batching affect latency and throughput?

Batching can improve throughput by amortizing fixed overhead across multiple items, such as a network round trip or database operation. However, an item may have to wait until enough other items arrive to form a batch. Larger batches can also consume more memory and create longer individual operations. The appropriate batch size therefore depends on the workload and the required latency target.

A Java API has good average latency but poor p99 latency. What would you investigate?

I would first establish whether the tail is caused by queueing, downstream dependencies, garbage collection, lock contention, thread-pool saturation, or connection-pool exhaustion. I would correlate p99 request latency with resource and dependency metrics over the same workload. I would also inspect traces for slow downstream calls and look for bursts or uneven workload distribution. Fixing the tail usually requires identifying the specific bottleneck rather than simply adding more application threads.

How would you decide whether to increase concurrency in a Java backend service?

I would measure the current workload and determine whether the service is CPU-bound, I/O-bound, or constrained by a downstream resource. Then I would increase concurrency experimentally while monitoring throughput, latency percentiles, CPU, memory, thread-pool queues, connection pools, and downstream behavior. If throughput stops increasing while latency continues rising, the system has likely reached or exceeded a bottleneck. I would then address the bottleneck or tune concurrency rather than continuing to increase it.