Track gRPC Service Errors

gRPC's binary protocol and streaming capabilities make it ideal for microservices, but its error handling model differs significantly from REST. Unlike HTTP where errors map directly to status codes, gRPC uses a finite set of status codes and a separate details mechanism for rich error information. Proper gRPC error handling monitoring requires understanding this protocol-specific design and instrumenting your services to capture the full error context — status codes, error details, and traces across service boundaries.

As microservices adoption grows, the friction between gRPC's error model and traditional monitoring becomes clear. Most developers familiar with REST assume they can treat gRPC errors like HTTP; they can't. A status code of Unavailable could mean the backend crashed, the network dropped, or the service is overloaded — without additional context, you're flying blind. Adding interceptor-based monitoring and distributed tracing makes the difference between guessing and knowing.

Understanding gRPC Status Codes and Error Model

gRPC defines 16 status codes: OK, Cancelled, Unknown, InvalidArgument, DeadlineExceeded, NotFound, AlreadyExists, PermissionDenied, ResourceExhausted, FailedPrecondition, Aborted, OutOfRange, Unimplemented, Internal, Unavailable, and DataLoss. Unlike HTTP's broad 4xx/5xx categories, these map closely to the reason a request failed, not just the outcome.

// gRPC status codes are specific, not vague
// Unavailable = backend can't handle the request right now
// FailedPrecondition = request was well-formed but the resource state forbids it
// ResourceExhausted = quota hit, or server under load
// InvalidArgument = request validation failed

import "google.golang.org/grpc/codes"
import "google.golang.org/grpc/status"

// Return a specific error
if quota < 0 {
    return status.Errorf(codes.ResourceExhausted, "rate limit exceeded: have %d, need %d", quota, needed)
}
if user == nil {
    return status.Errorf(codes.NotFound, "user %d not found", userID)
}

The key insight: every error in gRPC is a status. Your handler either returns nil (success) or a status (error). This makes monitoring straightforward — intercept the status at the boundary.

Rich Error Details: Going Beyond Status Codes

A status code alone is rarely enough. gRPC allows you to attach structured error details to a status using the google.rpc.Status message. This is where you put machine-readable context: retry info, error locales, quota, and custom application details.

// error_details.proto
syntax = "proto3";
package google.rpc;

message Status {
  int32 code = 1;
  string message = 2;
  repeated google.protobuf.Any details = 3;
}

// In your service proto:
import "google/rpc/status.proto";
import "google/rpc/error_details.proto";

service OrderService {
  rpc CreateOrder(CreateOrderRequest) returns (Order) {}
}

import "google.golang.org/genproto/googleapis/rpc/errdetails"

// Attach structured details to a status
st := status.New(codes.FailedPrecondition, "inventory exhausted")
st, _ = st.WithDetails(
    &errdetails.Help{
        Hints: []string{"Check supplier availability", "Try again in 1 hour"},
    },
    &errdetails.RetryInfo{
        RetryDelay: durationpb.New(5 * time.Second),
    },
)
return st.Err()

On the client side, extract and inspect these details:

err := client.CreateOrder(ctx, req)
st, ok := status.FromError(err)
if ok && st.Code() == codes.FailedPrecondition {
    for _, detail := range st.Details() {
        if help, ok := detail.(*errdetails.Help); ok {
            log.Printf("Hints: %v", help.Hints)
        }
    }
}

This pattern scales: you define application-specific error details in proto, attach them server-side, and parse them client-side. Crucially, this metadata is visible to monitoring systems that inspect the status details.

Monitoring gRPC Errors with Interceptors

gRPC interceptors are middleware that wrap every RPC call. Unary interceptors handle request-response calls; stream interceptors handle streaming calls. By installing a monitoring interceptor, you capture every error at the service boundary.

import (
    "google.golang.org/grpc"
    "google.golang.org/grpc/status"
)

// Server-side unary interceptor: log all errors
func loggingInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    resp, err := handler(ctx, req)
    if err != nil {
        st, _ := status.FromError(err)
        log.WithFields(log.Fields{
            "method": info.FullMethod,
            "code":   st.Code(),
            "msg":    st.Message(),
        }).Error("RPC failed")
    }
    return resp, err
}

// Install it on your server
server := grpc.NewServer(
    grpc.UnaryInterceptor(loggingInterceptor),
)

For production monitoring, wire this into your observability platform. If you're using a Sentry SDK, the interceptor is where you'd capture the error event:

import "github.com/getsentry/sentry-go"

func sentryInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    resp, err := handler(ctx, req)
    if err != nil {
        st, _ := status.FromError(err)
        sentry.CaptureException(err)
        // Optionally add tags for gRPC context
        sentry.WithScope(func(scope *sentry.Scope) {
            scope.SetTag("grpc.method", info.FullMethod)
            scope.SetTag("grpc.code", st.Code().String())
        })
    }
    return resp, err
}

LightTrace accepts Sentry SDK events, so you can point your sentry.Init() at LightTrace by changing only the DSN:

sentry.Init(sentry.ClientOptions{
    Dsn: "https://<your-key>@light-trace.robomiri.com/1",
})

Every gRPC error is now fingerprinted, grouped by issue, and visible in your dashboard alongside stack traces and breadcrumbs.

gRPC Errors in Distributed Tracing

When errors cross service boundaries, you need distributed tracing to follow the call chain. Each gRPC call should create a span; if the call fails, the error detail must be logged in that span.

Using OpenTelemetry, the gRPC instrumentation library automatically creates spans for every RPC:

import (
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
    "google.golang.org/grpc"
)

// Install the OpenTelemetry gRPC interceptor
conn, err := grpc.Dial(
    "localhost:50051",
    grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
    grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()),
)
defer conn.Close()

server := grpc.NewServer(
    grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
    grpc.StreamInterceptor(otelgrpc.StreamServerInterceptor()),
)

Now every RPC is a span. If an RPC fails, the span is marked with the error:

Span: PaymentService.Charge
  Status: Error
  Error Code: ResourceExhausted
  Error Message: daily limit exceeded
  Duration: 145ms

When errors cascade across services, context propagation carries the trace ID through headers. A failed order in OrderService → failed payment in PaymentService → exhausted quota in LimitService shows as a single trace across all three services. This visibility is why cross-project tracing matters: if LightTrace can correlate traces across your gRPC services, a single dashboard shows you the full failure path.

Common gRPC Error Scenarios

Deadline exceeded: Client sets a timeout. Server doesn't respond in time. The status is DeadlineExceeded. Interceptors should log the deadline and elapsed time:

deadline, ok := ctx.Deadline()
if ok {
    elapsed := time.Until(deadline)
    log.Infof("deadline in %v", elapsed)
}

Unavailable backend: The network failed, or the backend crashed. gRPC automatically returns Unavailable. A good monitoring setup logs the address of the unreachable backend and the retry behavior.

Invalid argument: The client sent malformed data. The server returns InvalidArgument immediately. This is a client bug, not a server bug. Your dashboard should distinguish these (low severity) from Internal errors (high severity).

Resource exhausted: Rate limits, quota, memory pressure. The status is ResourceExhausted. Pair this with the quota details in error details so you know if it's a client abuse or genuine overload.

In distributed tracing, these errors appear as spans with error tags. Throughput vs. latency analysis shows whether errors correlate with load spikes: if ResourceExhausted errors spike when throughput jumps, you have a scaling problem.

Monitoring Performance and Error Rates

Beyond capturing individual errors, you want to track error rate as a metric. Success rate = (successful RPCs / total RPCs). If your error rate is 0.1% but it's all Unavailable, you have an infrastructure problem. If it's 2% of InvalidArgument, your client is broken.

gRPC interceptors can emit metrics to your observability backend:

import "go.opentelemetry.io/otel/metric"

var (
    rpcCounter      metric.Int64Counter
    errorCounter    metric.Int64Counter
)

func init() {
    meter := otel.Meter("myapp")
    rpcCounter, _ = meter.Int64Counter("grpc.rpc.total")
    errorCounter, _ = meter.Int64Counter("grpc.rpc.errors")
}

func metricsInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    rpcCounter.Add(ctx, 1)
    resp, err := handler(ctx, req)
    if err != nil {
        st, _ := status.FromError(err)
        errorCounter.Add(ctx, 1, attribute.String("code", st.Code().String()))
    }
    return resp, err
}

Push these metrics to your APM platform. LightTrace tracks transaction throughput and latency, so if you report these metrics, you can correlate error spikes with performance degradation. An Unavailable error combined with a latency spike suggests the backend is overloaded.

Best Practices for gRPC Error Handling

1. Be specific with status codes. Don't default to Unknown or Internal. Choose the code that describes why the call failed.

2. Attach error details. If your error is actionable (like RetryInfo or Help), include it in details so clients and monitoring systems can act on it.

3. Log context in every error path. User ID, order ID, resource ID — include anything that helps you debug the failure later. Pair this with microservices observability to tie errors back to a user or transaction.

4. Monitor interceptor-side. Don't rely on clients to report errors. Install interceptors on your servers to capture every error, even ones that crash the client.

5. Instrument streaming RPCs. Streaming calls are easy to overlook. Use stream interceptors to log errors and track streaming success rates.

6. Propagate context end-to-end. Use context propagation to ensure every gRPC call carries a trace ID. When a call fails in a downstream service, you can trace it back to the originating request.

7. Set alerts on error rate and status-code distribution. If ResourceExhausted errors exceed a threshold, page someone. If Internal errors suddenly appear, that's a bug — alert immediately.

LightTrace's alert rules let you threshold on error frequency and status code, delivered via email. Combined with distributed tracing and error fingerprinting, you can move from "something is broken" to "this specific code path in this specific service is failing" in seconds.

Bringing It Together

gRPC error handling is not a fire-and-forget afterthought. The protocol's structured error model and interceptor architecture make it possible to build precise, actionable observability. Status codes communicate why an error occurred. Rich error details give context. Interceptors capture errors at the service boundary. Distributed tracing connects errors across services. And metrics tie error rate to performance.

The combination of interceptor-based error capture, structured error details, and distributed tracing with an APM platform like LightTrace turns gRPC from a black box into a transparent, debuggable system. Every error is a fingerprinted issue. Every trace is a complete call path. Every metric tells a story.

If you're building a microservices system with gRPC, start with proper error handling today. The cost is low (a few interceptors, a DSN change); the return is a system you actually understand when things break.