Whether you're running a microservice, a distributed system, or a monolith, knowing what to measure is half the battle. The concept of golden signals in SRE monitoring comes from Google's Site Reliability Engineering book and describes four key metrics that reveal the health of any system: latency, traffic, errors, and saturation. These four measurements work together to give you a complete picture of whether your service is performing well or degrading.
Most teams struggle with observability overload—they instrument everything and end up drowning in dashboards. The golden signals provide a focused framework: measure these four dimensions, and you'll catch most problems before your users do.
What Are the Golden Signals?
The four golden signals are:
- Latency — How long requests take to complete. Track both the happy path (a successful request) and errors. A slow request is often worse than a failed one because it ties up resources.
- Traffic — The volume of requests hitting your system. Measured in requests per second, operations per second, or sessions per second depending on your workload.
- Errors — The rate at which requests fail. Count both explicit errors (5xx status codes) and implicit ones (wrong responses that clients can't handle).
- Saturation — How close your system is to its limits. This is usually CPU, memory, disk I/O, or database connections—whatever resource runs out first.
These four metrics are simple to understand but powerful in practice. They work because they focus on user-facing impact: if any of these four degrade, your users notice it.
Why Golden Signals Matter
Traditional metrics monitoring often focuses on the wrong things. Teams watch CPU usage, network bandwidth, and disk free space without connecting those to what users actually experience. You can have 10% CPU utilization and still be slow if your database is saturated. You can have zero errors and still lose customers if your API responds in 5 seconds.
The golden signals flip this around. They measure the observable impact on your service's users and clients. When you fix a golden signal problem, your users see the improvement immediately.
Golden signals form the foundation for error budgets and SLOs — once you know what your baseline is, you can define acceptable thresholds and burn budgets against them.
Monitoring Latency Effectively
Latency is often split into percentiles: p50, p75, p95, and p99. The median (p50) is useful but misleading—a few slow requests can tank your p99, and that's often where your slowest customers live.
Track latency at different layers: frontend JavaScript, your application code, database queries, and external API calls. A slow endpoint might be fast in your app but slow waiting for a payment processor. Use distributed tracing to understand where latency is spent. When a request takes 2 seconds, is it 1.5 seconds in the database and 0.5 seconds in your app? Or 1 second waiting for a downstream service? Tracing answers these questions without guessing.
The key insight: p99 latency tells you how your worst-off users experience the system. If p99 is 5 seconds while p50 is 100ms, you've got a cohort of users suffering while most see acceptable performance.
Managing Traffic and Capacity
Traffic metrics tell you whether your system is being used normally or experiencing a spike. A sudden drop in traffic might indicate a DNS issue or client-side failure. A spike could signal a viral feature or a resource exhaustion problem.
Measure traffic by endpoint, by user segment, or by deployment. This helps you correlate traffic patterns with other signals—if errors spike exactly when traffic doubles, you've found your bottleneck. Understanding traffic also lets you forecast capacity: if traffic grows 20% month-over-month, you know when you'll need to scale.
Error Monitoring and Triage
Errors are the most straightforward golden signal. Count HTTP 5xx responses, exceptions in your application, timeouts, and failed external calls. But counting is only the start—you need to know which errors matter.
An error in your debug logging middleware is noise. An error in checkout is a business-critical incident. Implement error triage to separate signal from noise, and focus your on-call time on what breaks user workflows.
Group errors by root cause, not by surface symptoms. A NullPointerException could come from ten different code paths, but grouping them together reveals a pattern that a single occurrence would hide. This is where fingerprinting based on stack frames and context becomes critical.
Understanding Saturation and Resource Limits
Saturation is the hardest golden signal to define because it depends on your system. For a web server, it might be connection pools. For a database, it's query queue depth or lock wait time. For a background worker, it's message queue depth.
The key is to measure what will cause your system to fail first. If your API can handle 10,000 requests per second but your database saturates at 500, your real ceiling is 500 requests per second—not 10,000.
Saturation often precedes both latency and errors. When your database connection pool is 95% full, p95 latency is about to spike and errors are coming next. Monitor saturation proactively to alert before users are affected. Look for resource exhaustion in CPU, memory, disk I/O, thread pools, and database connections.
Don't confuse saturation with absolute resource usage. An 80% CPU reading with no latency increase is fine; 50% CPU with growing latency under constant load suggests saturation elsewhere (I/O, locks, contention).
Implementing Golden Signals Monitoring
Start simple: set up dashboards for each of the four signals. Use your application's native metrics (request counts, response times) plus infrastructure metrics (CPU, memory, I/O).
When latency increases, dive into distributed traces to pinpoint the culprit. When errors spike, filter by status code or exception type to find the root cause. When traffic doubles, check whether saturation increased proportionally—if not, your system scaled well. LightTrace captures all four signals: errors appear directly in your dashboard, latency and saturation surface through distributed tracing spans, and request volume is available through transaction metrics.
Set up alerts on thresholds (e.g., alert when p95 latency > 500ms or error rate > 1%), then build runbooks that tell on-call engineers what to check first. This is where error tracking best practices and structured logging combine with golden signals to create a complete incident response system.
The four golden signals aren't a complete observability solution, but they're the foundation. Once you're confident in these metrics, you can layer in deeper tracing, detailed logs, and domain-specific monitoring. But get these four right first, and you'll reduce MTTR dramatically.
Start tracking errors in minutes
Start monitoring your system's golden signals today with LightTrace. Track errors, latency, and transaction performance from any Sentry SDK—just point it at LightTrace and start shipping. Begin your free trial at light-trace.robomiri.com.
The beauty of golden signals is that they work at any scale, from a startup MVP to a global system handling billions of requests. They've stood the test of time because they measure what actually matters: whether your users can do what they came to do.