Prometheus Metric Types: Counter, Gauge, Histogram, Summary

If you're using Prometheus to monitor your systems, you'll need to choose between four metric types: Counter, Gauge, Histogram, and Summary. Each one answers a different question about your system's behavior, and using the wrong type will leave you unable to ask the questions that matter. This guide cuts through the confusion with concrete examples, the right PromQL queries, and the mistakes people make most often.

Prometheus metric types aren't just naming conventions—they determine what aggregations are legal, which functions you can use, and whether your dashboards will work at all. Choose wisely, and your metrics become a powerful lens into your system. Choose poorly, and you'll find yourself writing queries that don't make sense and debugging why your percentiles look wrong.

Counter

A counter is a monotonically increasing value. It only goes up (or resets on service restart). Use counters for anything you want to accumulate: total HTTP requests, bytes transmitted, errors seen, jobs completed, database queries run.

Here's a Go example using the Prometheus client library:

package main

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
)

var (
	httpRequests = promauto.NewCounterVec(
		prometheus.CounterOpts{
			Name: "http_requests_total",
			Help: "Total HTTP requests",
		},
		[]string{"method", "path", "status"},
	)
	
	errorsTotal = promauto.NewCounterVec(
		prometheus.CounterOpts{
			Name: "errors_total",
			Help: "Total errors encountered",
		},
		[]string{"type"},
	)
)

// In your request handler
httpRequests.WithLabelValues("GET", "/api/users", "200").Inc()
httpRequests.WithLabelValues("POST", "/api/users", "500").Inc()

The critical insight: you almost never graph a raw counter. If you plot http_requests_total directly, you'll see a line that only climbs, which tells you very little. Instead, use rate() to compute the derivative over a time window:

rate(http_requests_total[5m])     # requests per second over the last 5 minutes
rate(errors_total[5m])             # errors per second over the last 5 minutes
increase(http_requests_total[1h])  # total requests in the last hour

When a Prometheus-monitored service restarts, its counters reset to zero. The rate() function is smart enough to handle this: it ignores the discontinuity and computes the rate correctly across the restart. This is why counters are the right choice for measuring throughput and error rates—they're resilient to the operational reality of restarts.

Gauge

A gauge is a snapshot of a value that can move up or down. Think of a speedometer or a thermometer: active connections, memory usage, queue depth, temperature, CPU cores available. Gauges have no direction bias—they just measure the current state.

Here's a Python example:

from prometheus_client import Gauge

active_connections = Gauge(
    'active_connections',
    'Number of active database connections',
    labelNames=['pool']
)

memory_usage_bytes = Gauge(
    'memory_usage_bytes',
    'Memory used in bytes',
    labelNames=['type']
)

# Update the gauges
active_connections.labels(pool='main').set(42)
memory_usage_bytes.labels(type='heap').set(536870912)

# If you need to increment/decrement
active_connections.labels(pool='main').inc()
active_connections.labels(pool='main').dec()

Querying gauges is different. You can ask for the current value or look at how it changed:

memory_usage_bytes                              # current value
rate(memory_usage_bytes[5m])                    # rate of change (not meaningful for most gauges)
delta(memory_usage_bytes[5m])                   # change over the last 5 minutes
avg_over_time(active_connections[5m])           # average over time
max_over_time(active_connections[1h])           # peak over the last hour

The dangerous mistake: if you model a cumulative value (like total transactions) as a gauge, you lose the ability to compute throughput. If you model a state value (like active connections) as a counter, you can't ask "how many connections right now?" The distinction matters because it changes which queries make sense.

Histogram

A histogram buckets observations into ranges and counts how many fell into each bucket. It's ideal for measuring distributions like latency: you define buckets like "under 10ms, under 50ms, under 100ms, under 500ms, under 1s" and Prometheus increments the appropriate bucket for each observation.

Here's a Go example measuring request latency:

var (
	requestDuration = promauto.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "request_duration_seconds",
			Help:    "Request latency",
			Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10},
		},
		[]string{"handler"},
	)
)

// Time a request
start := time.Now()
// ... handle request ...
requestDuration.WithLabelValues("/api/users").Observe(time.Since(start).Seconds())

Histograms automatically produce three metrics:

request_duration_seconds_bucket{handler="/api/users", le="0.01"}     # count < 10ms
request_duration_seconds_bucket{handler="/api/users", le="0.05"}     # count < 50ms
request_duration_seconds_bucket{handler="/api/users", le="+Inf"}     # total count
request_duration_seconds_sum{handler="/api/users"}                    # sum of all observations
request_duration_seconds_count{handler="/api/users"}                  # total observations

The buckets are cumulative: each bucket includes all smaller buckets. The label le (less than or equal) makes this explicit. This design is what enables percentile queries:

histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m]))   # 95th percentile latency
histogram_quantile(0.99, rate(request_duration_seconds_bucket[5m]))   # 99th percentile latency
histogram_quantile(0.50, rate(request_duration_seconds_bucket[5m]))   # median

The histogram_quantile() function is computed server-side by Prometheus (or your dashboard) using interpolation. This means quantiles are aggregatable across instances: you can query the 95th percentile across all instances, across a specific zone, or by handler, and it works correctly.

Summary

A summary is like a histogram, but it computes percentiles on the client side during instrumentation, rather than server-side. It produces _sum and _count metrics (like a histogram) plus pre-computed quantile metrics:

from prometheus_client import Summary

request_duration = Summary(
    'request_duration_seconds',
    'Request latency',
    labelNames=['handler']
)

with request_duration.labels(handler='/api/users').time():
    # ... handle request ...
    pass

A summary with quantiles=[0.5, 0.95, 0.99] produces:

request_duration_seconds_sum{handler="/api/users"}              # sum of observations
request_duration_seconds_count{handler="/api/users"}            # total count
request_duration_seconds{handler="/api/users", quantile="0.5"}  # median
request_duration_seconds{handler="/api/users", quantile="0.95"} # 95th percentile
request_duration_seconds{handler="/api/users", quantile="0.99"} # 99th percentile

The critical difference: summaries cannot be aggregated across instances. If you're running three instances of your service and you query request_duration_seconds{quantile="0.95"}, you're asking "which instance has data?" and getting back three separate 95th percentiles, not the true 95th percentile across all instances. This is why summaries are generally the wrong choice for distributed systems.

Choosing the Right Metric Type

Use this table as your decision guide:

The rule of thumb: if you care about percentiles in a distributed system, use a histogram. If you're measuring a counter-like accumulation (requests, errors), use a counter and query the rate. If you're measuring instantaneous state, use a gauge.

Cardinality: When Metric Labels Explode

Every unique combination of label values creates a new time series. A metric with three labels that each have 10 distinct values produces 1,000 time series. A metric with user_id (millions of values) or request_id will blow up your Prometheus storage and crash your dashboards.

This is why it's dangerous to label by high-cardinality attributes: user IDs, request IDs, session IDs, or URLs with embedded IDs all have cardinality measured in millions. Even seemingly innocent labels like client_ip can spiral. A single counter labeled by request path (which might have hundreds of values when URLs contain IDs) can easily create tens of thousands of series.

Follow this rule: use labels only for low-cardinality attributes like environment, handler, method, status code, or zone. Reserve high-cardinality queries for structured logging or traces, not metrics. This keeps Prometheus fast and your observability costs manageable.

Histograms vs. Summaries: The Aggregation Problem

This deserves emphasis because it's the most common gotcha. If you're building a microservices system and you instrument latency with a summary, you'll think you can query the 95th percentile of latency across your entire system. You can't. Each instance pre-computed its own 95th percentile, and you're looking at three separate 95th percentiles—not the true tail latency.

Histograms solve this because the buckets (with their cumulative structure) let the server compute accurate quantiles from raw data across all instances. This is why the Prometheus community and the SRE approach favor histograms for distributed systems.

Native and sparse histograms are newer approaches in Prometheus that offer better storage efficiency for very fine-grained buckets, but histogram semantics remain the same: buckets are cumulative, quantiles are server-side, and they aggregate.

Turning Metrics into Signals

Metrics tell you something is wrong. Errors and traces tell you what went wrong. Counters reveal when throughput drops or errors spike; gauges show you when resources are exhausted. Histograms surface when tail latencies degrade. Combined, they form the three pillars of observability—metrics, logs, and traces—each answering different questions. Use the right metric types, and you'll have the precision to build meaningful alerts and dashboards that actually guide your incident response.