Detect and Fix Goroutine Leaks in Go

When memory usage climbs steadily in a production Go service with no obvious explanation, the culprit is often goroutine leaks — tasks that never finish and hold onto resources indefinitely. Unlike C or Java, Go makes spawning concurrent work trivially easy with the go keyword, but that ease masks a real cost: stuck goroutines silently consume memory and file descriptors until your service crashes. Go goroutine leak detection through memory profiling is essential for any backend team running Go at scale.

This guide walks through the tools and patterns that surface goroutine leaks, explains why they happen, and shows you how to fix them — whether you're diagnosing an existing problem or building safeguards into your CI pipeline.

What are Goroutine Leaks and Why They Matter

A goroutine leak happens when a goroutine starts but never terminates. The goroutine continues to consume memory (its stack, local variables, and any closures it captures), hold open channel reads or writes, and block on system calls indefinitely. If you spawn 10,000 leaking goroutines, you accumulate 10,000 stacks; on a modern system with a 1 MB minimum stack per goroutine, that's 10 GB of memory doing nothing.

In long-lived services (API servers, workers, background jobs), a single routine goroutine leak can multiply into thousands of stuck tasks over hours or days. The symptom is gradual memory growth that no amount of garbage collection can recover, because the goroutines themselves are not garbage — they're still technically alive, waiting on a channel that will never send, or blocked in a syscall that never resolves.

The Symptoms: Detecting Memory Growth in Production

Before you can fix a leak, you need to know one exists. The first signal is often a dashboard alert: steady climb in memory (resident set size or heap size) over days, with no correlation to traffic spikes or code deploys. You restart the service, and memory climbs again.

If you have distributed tracing configured — such as pushing spans to LightTrace — you'll also observe that throughput metrics stay flat or even decline as goroutines accumulate, because the runtime is spending more CPU on scheduling and garbage collection. Transaction latency (p50, p75, p99) may creep upward as the GC pause times grow.

The simplest check is to log or expose runtime.NumGoroutine() on a periodic basis:

package main

import (
  "fmt"
  "log"
  "runtime"
  "time"
)

func main() {
  ticker := time.NewTicker(10 * time.Second)
  defer ticker.Stop()

  for range ticker.C {
    log.Printf("Goroutines: %d", runtime.NumGoroutine())
  }
}

If that number climbs monotonically and never descends, you have a leak. This is your starting pistol for investigation.

Using pprof to Profile Goroutines

Go's net/http/pprof package exposes a goroutine stack dump endpoint that lets you inspect exactly which goroutines are running and where they're blocked. Add it to any HTTP server:

package main

import (
  "log"
  "net/http"
  _ "net/http/pprof"
)

func main() {
  go func() {
    log.Println(http.ListenAndServe("localhost:6060", nil))
  }()

  // Your main application runs here
  select {}
}

Now you can hit /debug/pprof/goroutine to see a text dump, or use the go tool pprof CLI:

go tool pprof http://localhost:6060/debug/pprof/goroutine

Inside pprof, type top to see which functions appear in the most goroutine stack traces:

(pprof) top
Showing nodes accounting for 5234, 100% of 5234 total
      flat  flat%   sum%        cum   cum%
      5234  100%  100%       5234  100%  runtime.gopark
         0    0%  100%       5234  100%  main.leakyWorker
         0    0%  100%       5234  100%  main.main

In this example, all 5,234 goroutines are parked (blocked) inside leakyWorker. Use list leakyWorker to see the source:

(pprof) list leakyWorker
Total: 5234
ROUTINE ======================== main.leakyWorker in /path/to/main.go
    0      5234 (go)       0      5234 (go)
    1             func leakyWorker() {
    2             <-make(chan int) // Blocks forever; nobody closes it
    3             }

That's your leak: a channel that nobody ever sends on or closes. Every time you spawn this goroutine, it's trapped.

Runtime Monitoring with NumGoroutine

For continuous monitoring in production, expose runtime.NumGoroutine() as a metric that your observability platform scrapes. If you use Prometheus:

package main

import (
  "runtime"

  "github.com/prometheus/client_golang/prometheus"
)

var numGoroutines = prometheus.NewGaugeFunc(
  prometheus.GaugeOpts{
    Name: "go_goroutines",
    Help: "Number of active goroutines",
  },
  func() float64 {
    return float64(runtime.NumGoroutine())
  },
)

func init() {
  prometheus.MustRegister(numGoroutines)
}

Alert on sustained growth: if the 1-hour rate of change exceeds a threshold (e.g., "more than 100 new goroutines per minute"), fire an alert. This catches leaks early, before memory exhaustion. Many alert systems (like LightTrace's alert rules) let you set thresholds on custom metrics; use that.

Practical Debugging: Finding the Leak

Once pprof shows you the goroutines, take two snapshots separated by a few minutes:

go tool pprof http://localhost:6060/debug/pprof/goroutine > before.prof
# ... wait 5 minutes ...
go tool pprof http://localhost:6060/debug/pprof/goroutine > after.prof

Then compare them:

go tool pprof -base before.prof after.prof
(pprof) top -cum

This highlights which goroutines grew between the two snapshots. If leakyWorker jumped from 100 to 150 instances, that's your culprit.

Common leak patterns:

1. Goroutine blocked on receive from a channel that's never closed: The goroutine waits forever. Always ensure channels are closed by the sender, or use a context to signal shutdown.

2. Goroutine spawned inside a loop without a way to stop: If you spawn a goroutine for each incoming request but never clean up, the goroutines pile up. Use worker pools with bounded queues.

3. Forgotten sync.WaitGroup or context cancellation: You spawn goroutines to do work, but forget to wait for them or cancel their context, so they outlive the function that started them.

4. Time.After in a loop: Calling time.After() in a for loop without draining the old timer creates a new timer on each iteration. The old timers are never fired and leak:

// WRONG - leaks timers
for {
  select {
  case data := <-ch:
    process(data)
  case <-time.After(1 * time.Second): // Creates a new timer each loop
    timeout()
  }
}

// RIGHT - reuse a ticker
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for {
  select {
  case data := <-ch:
    process(data)
  case <-ticker.C:
    timeout()
  }
}

Automated Detection with goleak

Writing manual tests to catch goroutine leaks is tedious. The goleak package (by Uber) automates the check:

package worker

import (
  "testing"
  "github.com/stretchr/testify/require"
  "go.uber.org/goleak"
)

func TestWorkerNoLeak(t *testing.T) {
  defer goleak.VerifyNone(t)

  w := NewWorker()
  w.Start()
  // Do work...
  w.Stop()

  // goleak will fail the test if any goroutines are still running
}

Add goleak checks to your critical worker and server tests. CI will catch leaks before they reach production. Combine this with the thread pool exhaustion debugging patterns to prevent cascading failures.

Tying It Together with Distributed Tracing

Memory and goroutine issues rarely exist in isolation. When a backend service leaks goroutines, the latency and throughput usually suffer first — requests slow down, timeouts increase, and the error rate climbs. Distributed tracing surfaces this connection clearly.

With a platform like LightTrace, you'll see slow transactions and high tail latencies (p95, p99) before memory pressure causes a crash. The trace waterfalls show you which spans are stalling; combined with pprof snapshots at the same timestamp, you can correlate the spike in goroutines to a specific code change or traffic pattern. If you reduce API latency by fixing goroutine leaks, your transaction metrics improve directly.

LightTrace also captures errors and exceptions. If a goroutine leak causes file descriptor exhaustion, net.Dial will start failing with "too many open files" errors — those errors land in your LightTrace project, alerting you to the root cause.

Prevention and Monitoring Checklist

• Log runtime.NumGoroutine() every 10 seconds in development and production. Treat monotonic growth as a red flag.
• Expose goroutine count as a metric and alert on sustained growth (> 1% per minute over 1 hour).
• Use goleak in tests for all worker pools, servers, and services that manage goroutine lifecycles.
• Profile with pprof regularly, not just in crisis mode. Understand your goroutine baseline.
• Use contexts for cancellation instead of ad-hoc shutdown flags. context.WithCancel or context.WithTimeout makes cleanup explicit.
• Review timer and channel usage in loops. Reuse tickers, drain channels, and close senders.
• Correlate goroutine counts with trace metrics. When goroutine count rises, latency and GC pressure follow.

Goroutine leaks are preventable with the right tools and discipline. Use pprof to profile, expose metrics to measure, and add goleak tests to catch regressions. When you do find a leak, the fix is usually simple — close the channel, cancel the context, or drain the timer — but the debugging work upfront saves you from crisis debugging at 3 AM.