Debug Go Deadlocks with Race Detection

Deadlocks in Go concurrent code are notoriously hard to hunt down. Your goroutines get stuck waiting on channels or locks, your application hangs silently, and the runtime gives you little help—Go doesn't automatically detect all deadlocks the way some languages do. The combination of Go deadlock debugging concurrent channels select statements and lock ordering mistakes creates a class of bugs that can slip past development and testing, only appearing under specific load or timing conditions in production.

Unlike race conditions, which the Go race detector catches reliably, deadlocks often manifest as mysterious hangs where the root cause is buried in goroutine stack traces. This guide covers how to identify stuck goroutines, use runtime tooling to surface deadlock patterns, and integrate distributed tracing to correlate goroutine hangs with real user impact.

The Silent Killer: Why Go Doesn't Detect Deadlocks

Go's runtime scheduler is optimized for throughput and simplicity. It does not employ cycle-detection algorithms to spot circular wait conditions or lock-ordering violations—behaviors that would add overhead to every context switch. If all runnable goroutines are waiting (on channels, mutexes, or network I/O) and nothing is waking them, the entire program hangs.

The key difference from race conditions: a race detector can instrument memory reads and writes and flag unsynchronized access. A deadlock detector would need to track lock-acquisition order globally across millions of goroutines, which is impractical. Go's philosophy puts the responsibility on you.

Common deadlock patterns in Go include:

• Unbuffered channel waits: A goroutine sends on a channel with no receiver, or vice versa.
• Lock ordering: Thread A locks mutex X then waits for Y; thread B locks Y then waits for X.
• Select statement deadlock: All branches of a select block on channel operations, and no channel is ready.
• Goroutine leaks masking deadlocks: A goroutine gets stuck, and no one notices until resource limits are hit.

Detect Deadlocks with pprof and SIGQUIT

Your first tool is the Go runtime's built-in goroutine dump. Send SIGQUIT to a running Go process:

killall -QUIT myapp

This writes a full goroutine stack trace dump to stderr. Look for goroutines in chan send or chan receive states, especially if many are stuck at the same line:

goroutine 42 [chan send]:
main.sender(0xc0001b2340)
        /app/main.go:18 +0x44
created by main.main
        /app/main.go:12 +0x08

If the goroutine count keeps growing while the program hangs, you have a goroutine leak combined with a deadlock.

The go-deadlock Library: Automated Detection

For development and staging, the go-deadlock library wraps sync.RWMutex and sync.Mutex to detect potential deadlocks before production:

go get github.com/sasha-s/go-deadlock

Replace your mutexes in test code:

package main

import (
	"github.com/sasha-s/go-deadlock"
)

type Account struct {
	mu      deadlock.RWMutex
	balance int
}

func (a *Account) Withdraw(amount int) bool {
	a.mu.Lock()
	defer a.mu.Unlock()

	if a.balance >= amount {
		a.balance -= amount
		return true
	}
	return false
}

When a mutex is held longer than the configured threshold (default 5 seconds), or when lock-ordering violations occur, go-deadlock will panic and dump the offending goroutine stacks. This catches many deadlocks early.

The downside: go-deadlock adds CPU overhead and should not run in production. It's a testing and staging tool.

Diagnosing Channel Deadlocks with select Inspection

Many channel deadlocks happen in select blocks, where all branches are blocked:

func process(in, out chan int) {
	for {
		select {
		case val := <-in:
			result := expensive(val)
			out <- result  // If `out` is never drained, this blocks forever
		case <-ctx.Done():
			return
		}
	}
}

If the receiver of out is also waiting on in, you have a circular dependency.

Diagnosis steps:

1. Capture a goroutine dump (SIGQUIT).
2. Search for all goroutines stuck on the channel name ("out" or "in").
3. Check if the stuck goroutines form a cycle: A waits for B, B waits for A.
4. Add context deadlines: wrap channel operations with select cases for <-ctx.Done() or <-time.After(timeout).

func process(ctx context.Context, in, out chan int) {
	for {
		select {
		case val, ok := <-in:
			if !ok {
				return
			}
			result := expensive(val)

			// Prevent indefinite wait on out
			select {
			case out <- result:
			case <-ctx.Done():
				return
			}

		case <-ctx.Done():
			return
		}
	}
}

Timeouts are not a cure-all—they mask deadlocks instead of fixing them—but they prevent silent hangs and allow graceful shutdown.

Correlate Deadlocks with Distributed Tracing

When deadlocks occur in production, they show up as requests that hang or timeout. Ordinary request logs tell you the endpoint hung, but not why. Integrating distributed tracing gives you the span timeline and cross-service causality.

Instrument your Go service with a tracing library like OpenTelemetry or, simpler, send errors and performance data to LightTrace:

import (
	"github.com/getsentry/sentry-go"
	"time"
)

func init() {
	err := sentry.Init(sentry.ClientOptions{
		Dsn: "https://<key>@light-trace.robomiri.com/1",
	})
	if err != nil {
		log.Fatal(err)
	}
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
	hub := sentry.GetHubFromContext(r.Context())
	
	span := hub.Span()
	if span == nil {
		span = sentry.StartSpan(r.Context(), "http.server")
	}
	defer span.Finish()

	// Your handler logic
	result := processWithTimeout(r.Context(), 5*time.Second)
	w.Write(result)
}

func processWithTimeout(ctx context.Context, timeout time.Duration) []byte {
	ctx, cancel := context.WithTimeout(ctx, timeout)
	defer cancel()

	done := make(chan []byte, 1)
	go func() {
		done <- expensiveWork(ctx)
	}()

	select {
	case result := <-done:
		return result
	case <-ctx.Done():
		sentry.CaptureException(ctx.Err())
		return nil
	}
}

When a request times out, LightTrace captures the error context and lets you examine the stack trace and breadcrumbs to pinpoint which goroutine or lock was blocking.

Prevention: Best Practices for Concurrent Go

1. Always use context cancellation: Pass context.Context through your call chain. Respect cancellation signals in select blocks.

2. Buffer channels thoughtfully: Unbuffered channels are a common deadlock source. Ask: "What if the sender and receiver are blocked on each other?"

3. Establish lock-ordering rules: If your code must lock multiple mutexes, document the order globally and enforce it everywhere. Use tools like go vet extensions or code review discipline.

4. Use defer for unlock: Prevent forgotten unlocks that leave goroutines waiting indefinitely.

5. Test with -race flag: While the race detector won't catch deadlocks, it will catch data races that often correlate with deadlock-prone code. Run your test suite with go test -race ./....

6. Monitor goroutine count in production: Export goroutine counts via a metrics endpoint. A climbing count signals leaks or deadlocks forming.

Tooling Integration: From Development to Production

• Local testing: Use go-deadlock in unit and integration tests. Run with -race and a reasonable timeout per test.
• Staging: Deploy with pprof endpoints enabled. Use killall -QUIT or HTTP goroutine dumps to inspect live behavior under realistic load.
• Production: Export goroutine and lock contention metrics. When request latency spikes, correlate it with goroutine count. Integrate error and trace collection so timeouts and context-cancellation errors are logged with full stack context.

LightTrace's integration with Go services means slow requests and timeout errors are automatically grouped by stack trace and linked to source code on GitHub. When a handler logs a context deadline exceeded error, you see which exact line in your concurrent code timed out—often the smoking gun for a deadlock.