gRPC RST_STREAM Errors: HTTP/2 Resets Explained

When your gRPC client logs something like stream terminated by RST_STREAM with error code: 2 or PROTOCOL_ERROR, you're seeing an HTTP/2 frame interrupt — not a gRPC error. That distinction matters enormously: gRPC runs on top of HTTP/2, so a gRPC RST_STREAM error lives at the transport layer, where the connection itself is telling you the stream is dead. Your gRPC status codes (0 through 16) are a completely separate numbering system. Understanding the difference between the two — and where to look for the root cause — saves hours of wild debugging in production.

This guide walks you through what RST_STREAM actually is, the HTTP/2 error codes you'll see in logs, the most common causes, and how to diagnose them.

What RST_STREAM is (and isn't)

HTTP/2 multiplexes multiple streams (requests) over a single TCP connection. Each stream has its own flow control, error handling, and lifecycle. When something goes wrong mid-stream, HTTP/2 sends a RST_STREAM frame — a 9-byte signal that terminates that stream alone without killing the whole connection.

The error code in RST_STREAM is an HTTP/2 error code, not a gRPC status. So error code: 2 means HTTP/2 INTERNAL_ERROR, which is completely different from gRPC status code 2 (UNKNOWN). This confusion is the root of most false leads in production debugging.

A gRPC status lives in the response trailers (metadata sent after the body). RST_STREAM is a raw HTTP/2 frame that lands before the trailers can be sent — it's the TCP/HTTP/2 layer saying "I'm shutting this stream down immediately." This is why you see RST_STREAM in logs from network tools (tcpdump, Wireshark, proxy access logs) but not in your gRPC client's status details.

HTTP/2 error codes: the reference table

When you see RST_STREAM with code 8 (CANCEL) from the client side, that is almost always intentional — the user hit cancel, the timeout fired, or the context was closed. Not a bug. Codes 3, 6, 11, and 12 are almost always configuration or proxy issues. Codes 2 and 9 point at the server.

Common causes and how to fix them

Client cancelled the call (HTTP/2 error 8 — CANCEL)

When a gRPC client calls context.cancel() or hits a deadline, it sends RST_STREAM with error code 8. This is normal — not a bug, not a server problem. It appears in logs because the server sees the stream die abruptly.

Fix: If clients are cancelling unexpectedly, look at timeout configuration on the client side. See connection timeout vs read timeout for the distinction; gRPC usually respects the context deadline.

Proxy or load balancer idle timeout (various codes)

A reverse proxy, load balancer, or firewall between client and server silently closes idle connections. When a long-lived bidirectional stream (like gRPC streaming) goes quiet for longer than the proxy's timeout (often 30–60 seconds), the proxy tears down the stream with RST_STREAM, usually code 7 (REFUSED_STREAM) or code 1 (PROTOCOL_ERROR).

Fix: Configure keepalive pings on the client side to send heartbeats over idle streams. In most gRPC SDKs:

grpc.keepalive_time_ms = 30000       # Ping every 30s
grpc.keepalive_timeout_ms = 5000     # Wait up to 5s for pong
grpc.keepalive_without_calls = true  # Ping even when idle

This tells the client to ping the server regularly, resetting the idle timer at the proxy.

Max concurrent streams exceeded (HTTP/2 error 7 — REFUSED_STREAM)

Every HTTP/2 connection has a SETTINGS_MAX_CONCURRENT_STREAMS limit (often 100–1000 streams per connection). When a client tries to open more streams than this limit, the server sends RST_STREAM with code 7.

Fix: Either increase the server's MAX_CONCURRENT_STREAMS setting (if you control it) or pool connections better on the client side to avoid opening too many streams on a single connection. For load testing or batch workloads, consider multiple connections.

Keepalive ping misconfiguration (HTTP/2 error 11 — ENHANCE_YOUR_CALM)

This is the production classic. If a client sends keepalive pings too aggressively (e.g., every 1 second instead of every 30 seconds), the server's keepalive enforcer fires back a GOAWAY frame with reason ENHANCE_YOUR_CALM (code 11) and closes the connection, often triggering RST_STREAM on active streams.

Fix: Ensure your keepalive interval is sensible. Most servers enforce a minimum (5–10 seconds). If you see this in logs:

grpc.http2.min_time_between_pings_ms = 10000   # Server rejects pings under 10s
grpc.http2.max_pings_without_data = 0          # Allow pings during idle (0 = unlimited)

And on the client, match the server's minimum:

grpc.keepalive_time_ms = 15000       # At least 15s apart, often 30s or more

Max header list size exceeded (HTTP/2 error 6 — FRAME_SIZE_ERROR)

If request or response metadata is unusually large (many headers, long strings), the sender might exceed the receiver's MAX_HEADER_LIST_SIZE limit. HTTP/2 responds with code 6.

Fix: Reduce metadata size, or increase the receiver's limit:

grpc.http2.max_frame_size = 16384          # Default, rarely needs change
grpc.max_header_list_size = 8192           # Increase if large headers

In gRPC services, avoid putting huge strings into context metadata or response headers.

Flow control stall (HTTP/2 error 3 — FLOW_CONTROL_ERROR)

In bidirectional or server-initiated streaming, if the sender sends data faster than the receiver advertises it can handle (via HTTP/2 WINDOW_UPDATE frames), the sender violates flow control. The receiver resets with code 3.

Fix: This is rare in well-behaved gRPC services, but if you see it, it usually means the receiver is slow or hanging. Check for blocking I/O or stuck goroutines on the receiver side. Increase the receiver's window size if this is a high-throughput bidirectional stream:

grpc.http2.initial_connection_window_size = 1048576   # 1 MB instead of default

HTTP/1.1-only proxy in the path (HTTP/2 error 13 — HTTP_1_1_REQUIRED)

Some very old proxies or firewalls don't understand HTTP/2 and will respond to an HTTP/2 upgrade with a 426 Upgrade Required. The server then sends RST_STREAM with code 13.

Fix: Either upgrade the proxy, configure it to pass through HTTP/2 CONNECT, or fall back to HTTP/1.1 + gRPC on HTTP/1.1 (if your SDK supports it). This is rare on modern infrastructure.

How to debug RST_STREAM errors

Enable HTTP/2 frame logging

Most gRPC SDKs support frame-level debugging. For gRPC Go:

GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info ./your_app

For Java with Netty:

-Dio.grpc.netty.shaded.io.netty.handler.ssl.CipherSuiteUtils.DEFAULT_TLSV13_CIPHER_SUITES=ENABLED \
-Dgrpc.verbosity=debug

This will show RST_STREAM frames in stderr, including the error code and which stream was reset.

Check proxy and load balancer logs

If you're behind a proxy, the real clue is often in the proxy's access log, not your app logs. Look for idle timeouts, stream limits, or connection resets:

proxy access log: 127.0.0.1 - - [time] "POST /helloworld.Greeter/SayHello HTTP/2" 200 15 - "-" 0.125
proxy error log: stream reset (error code: 7, max concurrent streams: 100)

Correlate with deploys

RST_STREAM errors that suddenly spike often follow a deploy. Check:

1. Did client keepalive config change?
2. Did server keepalive enforcement change?
3. Did proxy SETTINGS or timeouts change?
4. Did a new version of the gRPC library ship?

Use distributed tracing to see the full context

A distributed trace from client through proxy to server shows exactly where the stream died. If the trace stops mid-request, the RST_STREAM hit before the server could finish the response. If it completes, RST_STREAM was sent after the trailers — possibly the client closing the connection after receiving the response.

Putting it together

The next time you see stream terminated by RST_STREAM, pause and ask:

1. What is the error code? Look it up in the table above.
2. When did it start? Correlate with deploys, config changes, or traffic spikes.
3. Where is it happening? Client-side cancellation (8), proxy idle timeout (1 or 7), server overload (7), server crash (2), or keepalive misconfiguration (11)?
4. Is this normal? Code 8 (CANCEL) is almost never actionable. Code 11 (ENHANCE_YOUR_CALM) is almost always a config typo.

Once you know the layer and the cause, the fix is usually a one-line config change or a deploy to the right component. And if you're running gRPC error monitoring, you'll have the stack trace, span context, and user info to hand the right team immediately.