Circuit Breaker Pattern: Stopping Cascading Failures

The circuit breaker pattern is a fault tolerance mechanism that prevents cascading failures in distributed systems. When a downstream service fails or becomes sluggish, the circuit breaker stops sending requests to it, failing fast instead. This protects upstream services from being dragged down by a single failing dependency and gives the broken service time to recover.

Circuit breaker trips manifest as error spikes in your error tracking system. In LightTrace, you'll see the moment a service becomes unhealthy, the circuit opens to protect the rest of your architecture, and then recovers. Understanding these state transitions is critical to diagnosing production issues quickly and preventing the cascade effect that takes down an entire platform.

Understanding the Circuit Breaker Pattern

The circuit breaker pattern operates in three distinct states:

Closed state — Normal operation. The breaker monitors outbound requests, counting failures and measuring latency. When either metric exceeds a threshold, it transitions to open.

Open state — All requests fail immediately or return a fallback response without attempting to reach the downstream service. The breaker stops hammering an unhealthy service, giving it time to recover. After a configurable timeout, it moves to half-open.

Half-open state — The breaker permits a small number of test requests to the recovering service. If those requests succeed, the breaker returns to closed. If they fail again, it reopens.

Each state transition is a critical event. When you see a cluster of errors spike in LightTrace, your error-triage-process should immediately ask: Is a downstream service failing, or is a circuit breaker protecting us? Both are important signals, but they demand different responses.

Why Cascading Failures Happen Without Circuit Breakers

In distributed systems, failure propagates like a chain reaction. Service A calls Service B, which calls Service C. If Service C crashes, Service B's requests to it start timing out. Those timeouts consume threads. Service B exhausts its thread pool waiting for unresponsive connections, then begins rejecting requests from Service A. Service A's thread pool fills up next. Within seconds, all three services are down.

This is the cascading failure: one broken service brings down the entire stack. Circuit breakers interrupt this chain. When Service B detects that Service C is failing, its circuit breaker opens. Service B stops sending requests to Service C immediately, instead returning a cached response or a sensible default. Service A never sees a timeout; Service B responds instantly with a fallback. The rest of the system stays healthy while you fix Service C.

This aligns with error budgets and SLOs: a circuit breaker trades a small, controlled failure (one service unavailable) for system-wide resilience.

Implementing a Circuit Breaker

Here's a practical, focused implementation in Node.js:

class CircuitBreaker {
  constructor(fn, options = {}) {
    this.fn = fn;
    this.failureThreshold = options.failureThreshold || 5;
    this.resetTimeout = options.resetTimeout || 60000; // 60 seconds
    this.state = 'closed'; // closed, open, half-open
    this.failures = 0;
    this.lastFailureTime = null;
    this.successCount = 0;
  }

  async call(...args) {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        this.state = 'half-open';
        this.successCount = 0;
      } else {
        throw new Error('Circuit breaker is open');
      }
    }

    try {
      const result = await this.fn(...args);
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }

  onSuccess() {
    this.failures = 0;
    if (this.state === 'half-open') {
      this.successCount++;
      if (this.successCount >= 2) {
        this.state = 'closed';
      }
    }
  }

  onFailure() {
    this.failures++;
    this.lastFailureTime = Date.now();
    if (this.failures >= this.failureThreshold) {
      this.state = 'open';
    }
  }
}

// Usage
const fetchUserService = new CircuitBreaker(
  async (userId) => {
    const response = await fetch(`https://user-service.example.com/users/${userId}`);
    if (!response.ok) throw new Error(`HTTP ${response.status}`);
    return response.json();
  },
  { failureThreshold: 5, resetTimeout: 60000 }
);

async function getUser(userId) {
  try {
    return await fetchUserService.call(userId);
  } catch (err) {
    if (err.message === 'Circuit breaker is open') {
      // Return cached or default response
      return { id: userId, name: 'User (cached)', status: 'unavailable' };
    }
    throw err;
  }
}

This breaker transitions through states and returns immediately when open, avoiding the slow-death scenario of cascading timeouts.

Fallback Strategies

A circuit breaker alone is incomplete. You also need a fallback strategy for when the breaker opens:

Cached responses — Return recent data from cache if the service is down. Works well for read-heavy operations and non-critical data.

Default values — Return safe, minimal defaults (e.g., an empty list instead of calling an inventory service, a base recommendation instead of fetching personalized ones).

Graceful degradation — Disable non-essential features. If a recommendations service is down, show the store without personalization instead of failing entirely.

Fail fast — Return an error immediately without waiting for timeout. Users get a fast error and can retry, rather than hanging for 30 seconds.

In structured-logging-best-practices, log which fallback was triggered and why. This context is invaluable when diagnosing why users experienced partial outages.

Monitoring Circuit Breaker State Transitions

State transitions should flow into your error tracking. LightTrace captures errors automatically, but you should also emit explicit signals when the breaker opens or closes:

const Sentry = require('@sentry/node');

Sentry.init({
  dsn: 'https://<key>@light-trace.robomiri.com/1',
  environment: 'production',
});

class MonitoredCircuitBreaker extends CircuitBreaker {
  onFailure() {
    super.onFailure();
    if (this.state === 'open') {
      Sentry.captureMessage(
        `Circuit breaker opened: downstream service unavailable`,
        'warning'
      );
      Sentry.setTag('circuit_breaker_state', 'open');
      Sentry.setTag('service', 'user-service');
    }
  }

  onSuccess() {
    super.onSuccess();
    if (this.state === 'closed' && this.successCount === 1) {
      Sentry.captureMessage(
        `Circuit breaker recovered`,
        'info'
      );
      Sentry.setTag('circuit_breaker_state', 'closed');
    }
  }
}

Set up alert rules in LightTrace to notify you when circuit breakers open. A single open breaker is a symptom; if it's happening frequently or across multiple services, you have a deeper issue to investigate. This is where reduce-mttr becomes critical: fast detection and alerting shrink the time you spend blind to outages.

Testing Circuit Breaker Behavior

Test all state transitions:

it('opens after failure threshold is reached', async () => {
  const breaker = new CircuitBreaker(
    () => Promise.reject(new Error('downstream fail')),
    { failureThreshold: 2, resetTimeout: 100 }
  );

  for (let i = 0; i < 2; i++) {
    try {
      await breaker.call();
    } catch (err) {
      // Expected
    }
  }

  expect(breaker.state).toBe('open');

  // Subsequent calls fail immediately
  await expect(breaker.call()).rejects.toThrow('Circuit breaker is open');

  // After timeout, transitions to half-open
  await new Promise(r => setTimeout(r, 150));
  expect(breaker.state).toBe('half-open');
});

Testing ensures your fallback logic activates as intended and that timeouts trigger state transitions correctly.

Circuit Breakers in Distributed Systems

Use the circuit breaker pattern wherever you have an outbound request with non-negligible failure risk:

• Microservice calls over the network
• External API integrations (payment processors, analytics, third-party services)
• Database queries prone to hotspots or connection pool exhaustion
• Cache layer failures where aggressive retries would worsen the problem

Many languages have production-ready libraries: resilience4j (Java), Polly (.NET), tenacity (Python), hyper (Rust). Popular HTTP clients like axios and got have circuit breaker middleware. Start simple, then tune thresholds and timeouts based on your production metrics.

Pair circuit breakers with observability: use metric-types-explained to track state transitions, duration in each state, and fallback invocations. This data reveals whether your thresholds are too tight (circuit opens too often) or too loose (service stays degraded too long).

The circuit breaker pattern is a proven way to architect resilient systems that degrade gracefully instead of failing catastrophically. When you combine it with error tracking, structured logging, and alert rules, you see failure coming and contain it before it spreads.