gRPC "Failed to Pick Subchannel": Diagnose and Fix

When your gRPC client hits UNAVAILABLE: failed to pick subchannel, or you see Name resolution failed, your service is trying to connect but can't find a healthy backend to send traffic to. This error is frustratingly vague — it could mean the DNS resolver returned nothing, all your backends are down, TLS is broken on every instance, or the client gave up before the connection was ready. This guide walks you through the model, the most common causes, and how to debug it methodically.

How gRPC connection picking works

To fix "failed to pick subchannel," you need to understand what a subchannel is and why the picker is rejecting all of them.

A gRPC channel is a virtual connection to a remote service. When you create a channel with a target like dns:///my-service:50051, the resolver (DNS in this case) translates that target name into a list of concrete IP addresses. Each address becomes a subchannel with its own TCP connection and connectivity state.

Every subchannel moves through states:

• IDLE — not yet connected, waiting to be used
• CONNECTING — actively trying to establish a connection
• READY — successfully connected and passing health checks (if health checking is enabled)
• TRANSIENT_FAILURE — a temporary error (e.g., connection refused, TLS handshake failed)
• SHUTDOWN — closed or error state

The load-balancing policy (like pick_first or round_robin) uses these states to decide which subchannel gets your RPC. The error "failed to pick subchannel" means the policy had no READY subchannel to pick from.

Causes and diagnosis

DNS returned no addresses or resolver failed

Your resolver tried to look up the target hostname and got nothing — either a DNS NXDOMAIN error, or the endpoint isn't registered yet.

Check DNS immediately:

# For a Unix socket or IPv4 address, skip this. For a DNS name:
nslookup my-service
# or
dig my-service
# In Kubernetes:
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup my-service.default.svc.cluster.local

Fix:

• Verify the target name is spelled correctly in your channel URI (dns:///my-service:50051).
• Ensure DNS is up and your domain is registered. For Kubernetes, check that the DNS name resolves (e.g., my-service.default.svc.cluster.local).
• If using a headless service, ensure it's created first.

All backends are down or in TRANSIENT_FAILURE

The resolver found addresses, but every subchannel failed to connect or is still connecting.

Check with gRPC channelz (programmatic connectivity inspection):

import "google.golang.org/grpc/channelz"

// Print the channel's connectivity state and subchannels
channelz.GetTopChannels(0)
// or for a specific channel:
channelz.Entry(channel.Target())

Or enable verbose gRPC logging to see state transitions:

# Go
GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_VERBOSITY=DEBUG GRPC_TRACE=channel ./your_app

# Java
-Djava.util.logging.config.file=logging.properties
# In logging.properties: io.grpc.level=FINE

# Python
import logging
logging.basicConfig()
logging.getLogger('grpc').setLevel(logging.DEBUG)

The logs will show each subchannel attempting to connect and either succeeding (READY) or failing (TRANSIENT_FAILURE). Look for the actual error: connection refused, timeout, or TLS failure.

Fix:

• Verify the backend service is running. Restart it if necessary.
• Check that the port is correct and open. Use lsof -i :50051 (or your port) on the backend.
• Confirm the backend isn't firewalled from the client. Test with telnet my-service 50051 or curl -v https://my-service:50051 (gRPC is HTTP/2, so curl can probe it).

TLS handshake failing on every subchannel

If you're using TLS (most production setups do) and every subchannel enters TRANSIENT_FAILURE, the handshake is broken. This often happens when the server's certificate doesn't match the hostname the client is using.

Check certificate validity:

openssl s_client -connect my-service:50051 -servername my-service

Look for certificate CN (Common Name) or SAN (Subject Alternative Name) — does it match your target hostname?

Fix:

• Ensure the certificate's SAN or CN matches the hostname in your channel URI. If your URI is dns:///my-service:50051, the cert must cover my-service.
• If using mTLS, ensure your client certificate is valid and the server is configured to accept it.
• If you're in development and self-signed cert names keep changing, either pin the hostname you dial (make it match the cert) or disable certificate validation only locally (never in production).

Wrong target scheme or bare hostname

If you pass my-service:50051 instead of dns:///my-service:50051, gRPC might treat it as an IPv4 address and fail to resolve.

Check your channel URI:

• dns:///my-service:50051 — DNS resolver, any hostname
• ipv4:///192.168.1.1:50051 — static IPv4 address
• ipv6:///[::1]:50051 — static IPv6 address
• bare my-service:50051 — treated as an IPv4 literal, will fail if it's not a valid IP

Fix:

• Use dns:/// prefix for hostnames. Without it, gRPC tries to parse the string as an IP and fails.

Headless service / Kubernetes DNS returning nothing

In Kubernetes, a headless service (ClusterIP: None) returns A records pointing to each pod. If no pods match the label selector, DNS returns nothing.

# Check if the headless service has endpoints
kubectl get endpoints my-service
# Should list pod IPs; if empty, no pods matched the selector

Fix:

• Verify the pod labels match the service selector: kubectl get pods -L <label-key>.
• Ensure the pod is running and ready: kubectl get pods and check the READY column.
• Wait for the pod to reach Ready state (it must pass health checks if configured).

No backends passing health checks

If you've enabled gRPC health checking (recommended for production), subchannels only enter READY state if the backend reports healthy.

Check health check configuration:

• On the server: is the gRPC Health Checking protocol implemented? Most frameworks have a health service you must wire up.
• On the client: are you using a circuit breaker or load-balancer that queries health? Confirm it's enabled.
• Run a health check manually: grpcurl -plaintext my-service:50051 grpc.health.v1.Health/Check.

Fix:

• Implement the gRPC Health Checking protocol on the server (most frameworks provide this out of the box).
• Ensure the backend isn't returning SERVING or is reporting a failure reason.
• If health checks are timing out, increase the check interval or fix the underlying issue blocking health checks.

Client not waiting for the channel to become ready

By default, a gRPC call fails immediately if the channel isn't READY yet. If your client sends a call before the channel connects, you get "failed to pick subchannel" even if the backend is fine.

Check your RPC call:

// Default: fails fast if channel not ready
resp, err := client.MyMethod(ctx, req)

// With WaitForReady: waits for channel to become ready
resp, err := client.MyMethod(ctx, req, grpc.WaitForReady(true))

WaitForReady holds the call in a queue until a READY subchannel is available. The tradeoff: your call now waits (blocking), and a long deadline can mask slow startup.

Debugging with logs and metrics

Step 1: Enable verbose logging (see above for per-language setup). Look for lines like:

Subchannel created: 192.168.1.1:50051
Entering state CONNECTING
Entering state READY

If you only see TRANSIENT_FAILURE, check the reason: connection refused, TLS handshake failure, timeout, etc.

Step 2: Check the resolver output. For DNS:

# On the client machine
host -v my-service

If DNS works locally but not from your container/pod, you may have a network policy or DNS server issue.

Step 3: Verify backend health directly with grpcurl:

# Check if server is responding (no mTLS)
grpcurl -plaintext my-service:50051 list

# With TLS
grpcurl my-service:50051 list

# Health check
grpcurl -plaintext my-service:50051 grpc.health.v1.Health/Check

Step 4: Look at gRPC error codes. The log might also show intermediate errors like UNAVAILABLE (code 14) or DEADLINE_EXCEEDED (code 4). These are breadcrumbs leading to "failed to pick subchannel."

If you're building a resilient system, remember that transient cascading failure can happen: one overloaded backend brings down others. Implement health checks, circuit breakers, and fallback routes. External dependency failures are inevitable, so design for them.

For production systems, capture these connection errors in your observability stack so you catch them before users do. gRPC error handling should log every pick failure and its cause, tagged by service and subchannel.