When your API response time hits 2 seconds, should you wake up at 3 a.m.? Maybe. When your CPU spikes to 80%, should you page the team? Almost never. APM alerts — alerts on application performance metrics like latency, throughput, and error rate — sound simple but require discipline. Most teams trigger on the wrong signals, alert too quickly on noise, and end up ignoring all of them. A good performance alert means "a human must act now." Everything else is just data.
The difference between a production incident and false alarm is often the difference between alerting on user-visible symptoms and alerting on infrastructure details. This post walks through a decision framework for building APM alerts that actually matter.
Symptom vs. cause alerting
The cardinal rule: alert on symptoms, not causes. A symptom is what users see. A cause is what's happening under the hood.
Symptoms for web applications:
- High response latency (p95, p99)
- High error rate
- Low throughput / high response time combined
Causes:
- High CPU utilization
- High memory usage
- High disk I/O
- Garbage collection pauses
- Database connection pool exhaustion
The temptation is to alert on causes — they feel more "technical" and actionable. But a single-instance CPU spike to 90% might resolve in seconds (a temporary spike, garbage collection, a background job). A symptom alert on latency rising to 5 seconds tells you that users are experiencing a problem, which is what actually matters.
Avoid alerting on CPU, memory, or GC metrics in isolation. These metrics are noisy and rarely actionable without context. Alert when latency rises or error rate climbs — then investigate the CPU/memory afterward.
If you're using distributed tracing, trace your symptoms back to causes in your investigation. But your alert should fire on the symptom.
Static thresholds vs. baselines
A static threshold is simple: "alert if p95 latency exceeds 500ms." A baseline is dynamic: "alert if p95 latency is 2x your typical baseline for this time of day."
Static thresholds are crude but predictable. Everyone knows what "500ms" means. They're also easy to explain to non-technical stakeholders: "our API should respond in under a second." The downside is they don't adapt to your service's actual behavior. An endpoint that's always slow doesn't become a problem because you set a threshold; one that's suddenly slow but still under your static threshold gets missed.
Baselines sound better in theory — they adapt to reality — but they require reliable historical data and machine learning to avoid false positives. A rainy Tuesday morning might legitimately be slower than a sunny Friday afternoon (different user populations, different workloads), and anomaly detection algorithms struggle with that. They also introduce a hidden cost: you need to tune the sensitivity, which moves the problem from "pick a number" to "pick an algorithm."
Be honest about your limits: LightTrace implements new-issue and event-frequency thresholds, not anomaly detection. If you need baseline alerting, you'll combine LightTrace with another tool (like Datadog or New Relic) or accept static thresholds and tune them quarterly.
Start with static thresholds. Set them conservatively based on your SLA or your worst acceptable latency. After a few weeks of data, audit which alerts fire and adjust. Static thresholds are easier to reason about and fewer sources of false positives.
Duration and for-clauses: why "5 minutes" matters
An alert that fires the instant latency exceeds your threshold is hair-trigger and noisy. A transient spike — one burst of slow requests — isn't an incident. An alert that fires only after latency stays elevated for 5 minutes means something sustained is wrong.
The pattern in most monitoring systems is a "for" clause: "alert if p95 latency exceeds 500ms for 5 minutes."
What does "for 5 minutes" mean? It means the condition is true in every 1-minute window over the last 5 minutes. So if your p95 is 600ms for 4:00-4:01, 4:01-4:02, 4:02-4:03, 4:03-4:04, and 4:04-4:05, it fires. If it's 600ms for 4:00-4:01 and 4:04-4:05 but normal in between, it doesn't fire.
This matters because it filters transient noise while catching real incidents:
- A garbage collection pause might spike latency for 10 seconds. It won't sustain for 5 minutes. No alert.
- A database connection leak that compounds over time gets worse minute by minute. After 5 minutes, it fires.
- A deploy that gradually destabilizes throughput shows up as rising latency. It fires.
For very critical paths (payment processing, login), you might use 2 minutes. For background jobs, 10 minutes is fine. The rule: use the shortest duration that doesn't wake up your team for every restart.
Window size and traffic sensitivity
Percentiles are lies on low-traffic endpoints. If your endpoint serves 1 request per hour, one slow request gives you a p99 latency of "that one slow request" — it's useless as a threshold.
The sensitivity also changes by window size. A 1-minute window with 100 requests gives you 1-percentile resolution. The same endpoint at 1,000 requests per minute gives you 0.1-percentile resolution. The p95 of 1,000 requests is much more stable than the p95 of 100 requests.
When setting alerting thresholds, consider your traffic:
- High-traffic endpoints (1,000+ req/min): alert on p95 or p99, 1-2 minute window.
- Medium-traffic (100-1,000 req/min): alert on p95, 2-5 minute window. Avoid p99 — it's too noisy.
- Low-traffic (<100 req/min): alert on absolute latency (max response time) or error rate, not percentiles. Or disable APM alerts entirely and alert on error rate only.
If you're monitoring golden signals, latency is one of four; pay more attention to traffic and error rate for low-traffic paths.
Multi-window burn rates for SLO-based alerting
If you've defined an SLO (e.g., "99.5% of requests respond in under 500ms"), the right way to alert is not to alert on a fixed threshold. Instead, alert on how fast you're "burning" your error budget.
A fast-burn alert fires when you're consuming your error budget so fast you'll exhaust it in hours if it continues. A slow-burn alert fires when the burn rate is lower but sustained — you'll hit zero budget in a few days.
For example, with a 99.5% SLO:
- Fast burn (2% error rate / 40x SLO): alert after 5 minutes
- Slow burn (0.1% error rate / 2x SLO): alert after 1 hour
This beats a static threshold because it adapts to your SLO and scales with your traffic. A 2% error rate on a small endpoint is concerning; on a large endpoint processing millions of requests, it's expected noise.
LightTrace doesn't implement multi-window burn-rate alerting natively, but the concept is worth understanding when you design your alert strategy.
What NOT to alert on
Some signals are tempting but will flood you with noise:
| Signal | Why not | Better approach |
|---|---|---|
| Single-instance CPU | Restarts, autoscaling, and GC cause spikes that resolve instantly | Alert on latency instead |
| GC pause duration | Expected on JVM, Go, and .NET services; varies with workload | Alert when p95 latency rises |
| Individual slow transactions | One slow database query isn't an incident; millions are | Alert on p95/p99 latency or error rate |
| p99 on low-traffic endpoints | Meaningless percentile with <100 samples/minute | Use error rate or absolute latency |
| Request count | Not actionable unless correlated with latency or errors | Combine with error rate or latency |
Building an APM alert strategy
Start with golden signals: latency, error rate, throughput, and saturation. Alerting on saturation is infrastructure-level (disk, memory). Alerting on the first three is APM:
- Alert on latency for critical paths — p95 over 500ms (or your SLA) for 5 minutes.
- Alert on error rate — a spike from your baseline. LightTrace supports event-frequency thresholds, so set a rule like "alert if error rate for
GET /api/checkoutexceeds 5% over 5 minutes." - Skip throughput — rising request volume isn't an alert; rising latency with stable throughput is.
- Alert on saturation only if you own the infrastructure (CPU, disk). If you're on managed cloud, your provider handles it.
Route these alerts by service owner. Use mttx metrics to measure your detection speed — mean time to acknowledge, mean time to resolve. After a month, audit: which alerts were noise? Which ones you actually fixed?
LightTrace delivers alerts by email only, so route them to your on-call rotation and set email to interrupt-mode. If you need Slack or PagerDuty integration, you'll need to bridge via a webhook or third-party service.
Start tracking errors in minutes
Set up your first APM alert in LightTrace — monitor latency and error rate for your critical endpoints, and start tracking MTTD and MTTR to improve your incident response.
APM alerting is about cutting through noise to the signal that matters: users are waiting for your service, or it's returning errors. Every alert should earn its page. A sensible strategy starts with latency on your critical path, error-rate spikes on high-traffic endpoints, and a discipline to audit and disable alerts that don't lead to incidents. Read about error budgets and SLOs to understand how to tune thresholds to your business risk, and check out how-to-set-up-error-alerts for the mechanics of configuring rules in your monitoring system.