Golden Signals

The Golden Signals are four core service-level measurements: latency, traffic, errors, and saturation. They provide a compact framework for monitoring service health and deciding where deeper investigation should begin.

Latency measures how long requests take to complete, often tracked with distributions such as percentiles rather than only an average. For a Java backend, it can be useful to separate server-side processing time from downstream calls when diagnosing slow requests.

Traffic measures the amount of demand placed on a service. Depending on the system, useful units include requests per second, messages per second, jobs processed, or active connections. Traffic provides essential context for interpreting other metrics.

Errors measure unsuccessful or incorrect outcomes. They can include HTTP 5xx responses, failed RPCs, rejected requests, exceptions, or application-defined failures. An error rate is often more useful than a raw error count because it accounts for changing traffic volume.

Saturation describes how close a resource or service is to its practical capacity. CPU, memory, connection pools, thread pools, database pools, disk I/O, and queue depth can all reveal saturation. A service may still be returning successful responses while saturation is increasing, making it an important leading indicator.

The four signals complement one another. A latency increase during unchanged traffic may point toward a resource or dependency problem, while rising traffic with stable latency and errors may simply indicate increased demand. Alerts should focus on meaningful service impact and capacity risk rather than creating an alert for every individual metric change.

Key points

• Latency shows how quickly requests complete, preferably using distributions or percentiles.
• Traffic measures workload or demand on the service.
• Errors measure failed or incorrect request outcomes; rates often provide better context than counts.
• Saturation indicates how constrained a resource or service is relative to its useful capacity.
• The signals are most useful together because each supplies context for interpreting the others.

Interview questions

What does saturation mean, and why can it be an early warning signal?

Saturation describes how close a resource or service is to a useful capacity limit. Examples include a nearly exhausted thread pool, database connection pool, CPU capacity, or growing work queue. Saturation can rise before requests actually fail, so monitoring it can reveal capacity pressure before users see a large error increase.

How would you investigate a sudden increase in latency using the Golden Signals?

I would first check whether traffic also changed, because increased load can explain part of the latency increase. Then I would examine errors and saturation to determine whether failures, CPU, memory, queues, connection pools, or another constrained resource correlate with the event. I would also break latency down by endpoint, dependency, instance, or other useful dimensions to isolate the source.

What are the four Golden Signals?

The four signals are latency, traffic, errors, and saturation. Latency tells us how long operations take, traffic describes demand, errors show unsuccessful outcomes, and saturation indicates how close the system is to a limiting capacity. Together they provide a compact starting point for monitoring a service.

Why is average latency often insufficient for monitoring a backend service?

An average can hide a small but important population of very slow requests. Latency distributions and percentiles such as p95 or p99 expose tail behavior that users may experience even when the mean remains acceptable. The appropriate percentile depends on the service's workload and user-facing requirements.

How would you design alerts around the Golden Signals for a production Java service?

I would start from user-facing reliability objectives and choose metrics that represent meaningful service behavior. Alerts could combine sustained latency or error-rate breaches with relevant traffic and saturation context, while avoiding excessive sensitivity to short-lived fluctuations. I would also use separate diagnostic dashboards for dimensions such as endpoint, dependency, instance, thread pools, and database pools so that an alert leads quickly to investigation.