Dependency Failure Propagation

Dependency failure propagation describes how a failure in one component can spread to its callers and other dependent components through synchronous calls, shared resources, and retry behavior. Senior engineers mitigate it by containing blast radius, bounding resource consumption, and making failure behavior explicit.

In a distributed backend, service A may depend synchronously on service B. If B becomes slow or unavailable, A can accumulate blocked requests, consume threads or connection-pool entries, and eventually become unhealthy itself. Components depending on A can then experience the same problem, creating a cascade.

Latency propagation is especially dangerous because a dependency does not need to return errors to cause damage. A slow dependency can exhaust bounded resources such as servlet threads, HTTP connections, database connections, queues, or memory. Timeouts limit how long a caller waits, but poorly chosen timeouts can still create load spikes or synchronized retries.

Retries can amplify an outage when many callers retry the same failing operation. If each attempt consumes resources or reaches the same unhealthy dependency, the additional traffic can make recovery harder. Exponential backoff, jitter, retry limits, and selective retry policies reduce this amplification.

Containment techniques include strict timeouts, bounded concurrency, circuit breakers, bulkheads, load shedding, fallback behavior, and isolation of critical resources. A circuit breaker can temporarily stop calls to an unhealthy dependency, while a bulkhead prevents one dependency from consuming resources needed by unrelated work.

For a senior engineer, the key design question is not simply whether a dependency can fail, but how its failure changes resource usage and failure behavior throughout the call graph. Observability should expose dependency latency, timeout rates, error rates, retry counts, saturation, and circuit state so operators can identify propagation early.

Key points

• Synchronous dependency failures can propagate through blocked callers and exhausted resources.
• Slow failures can be as damaging as explicit errors because they consume bounded resources.
• Retries can amplify an outage unless attempts, backoff, and concurrency are bounded.
• Timeouts, circuit breakers, bulkheads, and load shedding provide different forms of containment.
• Trace dependency latency and resource saturation across the call graph to detect cascades.

Interview questions

How can a failure in one service cause unrelated services to fail?

A service often holds resources while waiting for a dependency, such as request threads or HTTP connections. If the dependency becomes slow, those resources remain occupied longer and concurrency rises. Once a shared pool is exhausted, the affected service may reject or delay other requests that do not use the failing dependency. Its callers can then experience the same symptoms, producing a cascade.

Why are retries dangerous during an outage?

A retry converts one failed operation into multiple attempts, so widespread retries can multiply traffic toward an already unhealthy dependency. This can consume more threads, connections, CPU, and network capacity and delay recovery. A robust retry policy normally limits attempts and uses exponential backoff with jitter. It should also distinguish transient failures from errors where another attempt is unlikely to help.

How would you design a Java service so a slow downstream dependency does not take down the service?

I would start with an explicit request deadline and bounded concurrency for downstream calls. The HTTP client and connection pool should have appropriate limits and timeouts so blocked calls cannot consume resources indefinitely. I would consider a circuit breaker, bulkhead isolation, selective retries with backoff and jitter, and load shedding where appropriate. Finally, I would instrument dependency latency, timeout rates, retries, pool saturation, and circuit state so the containment mechanisms can be validated under load.

What is the difference between a circuit breaker and a timeout, and why might you use both?

A timeout limits how long one operation waits for a response. A circuit breaker uses recent failure or latency observations to decide whether subsequent calls should be attempted at all. Using both can prevent individual calls from waiting indefinitely while also preventing repeated calls to a dependency that is already demonstrating sustained failure. The breaker should have a recovery mechanism, such as a half-open state that permits limited test traffic.

How would you decide whether a fallback is safe?

I would first define the correctness requirements of the operation rather than assuming that any response is better than an error. A fallback is appropriate when degraded data or behavior remains within an explicitly acceptable contract. For example, cached information may be acceptable for a read path but dangerous for a transaction requiring current state. The fallback itself must also have bounded resource use and should be observable so silent degradation does not become permanent.