Sidekiq Error Tracking: Catch Failing Background Jobs

Background job failures are invisible by default in Rails. You fire off a Sidekiq worker to send an email, charge a payment, or rebuild a cache—but if it crashes, no user sees a 500 error. There's no request to trace. The job just vanishes into worker logs, and by the time you notice it's gone, your queue is backed up and the context you needed to debug it is lost.

Sidekiq retries failed jobs with exponential backoff, which can mask the real problem. A broken job looks "fine" for hours as it retries in the background; then it lands in the dead letter queue and nobody catches it until a customer emails you three days later. Without proper sidekiq error tracking, these silent failures compound into data corruption, missed notifications, and broken workflows.

Sidekiq's Retry Model

Sidekiq retries failed jobs automatically. By default, a job that raises an exception will retry up to 25 times over several days, with exponential backoff and jitter to avoid thundering herd problems.

You can customize retry behavior per worker:

class ProcessPaymentJob
  include Sidekiq::Job
  sidekiq_options retry: 5  # Retry up to 5 times instead of 25
  
  def perform(order_id)
    order = Order.find(order_id)
    charge_card(order)
  end
end

Or disable retries entirely for jobs where idempotency is hard:

class SendOneTimeCodeJob
  include Sidekiq::Job
  sidekiq_options retry: false  # Don't retry; if it fails, it fails once
  
  def perform(user_id, code)
    User.find(user_id).send_sms(code)
  end
end

When retries are exhausted, Sidekiq moves the job to the Dead Letter Queue (morgue). By default, dead jobs stay there for six months or until you manually retry them. The problem: nobody checks the dead letter queue automatically, and logs tell you that the job died, but not why or what happened next in your system.

Error Handlers: Catching Every Exception

Sidekiq has a global error handler that fires on every job exception, including retries:

Sidekiq.configure_server do |config|
  config.error_handlers << lambda { |error, context|
    # Fires EVERY TIME a job raises, even on retries
    Rails.logger.error("Job failed: #{context[:job]['class']} - #{error.message}")
  }
end

This handler runs on every attempt, which is useful for logging and monitoring queue health. But if you're wiring this to LightTrace via the Sentry SDK, you typically want to be selective: alert on terminal failures, not transient retries that might succeed on the next attempt.

Sidekiq Retries Exhausted: The Real Alert

The hook you actually want is sidekiq_retries_exhausted, which fires only when a job fails permanently:

class ProcessPaymentJob
  include Sidekiq::Job
  sidekiq_options retry: 5
  
  def perform(order_id)
    order = Order.find(order_id)
    charge_card(order)
  end
  
  sidekiq_retries_exhausted do |msg, exception|
    # This fires ONLY when all 5 retries fail
    Sentry.capture_exception(exception, extra: {
      order_id: msg['args'][0],
      jid: msg['jid'],
      retry_count: msg['retry_count'],
    })
    
    # Optional: notify team, mark order as failed, etc.
    OrderFailureMailer.payment_failed(msg['args'][0]).deliver_now
  end
end

This is where you alert. A sidekiq_retries_exhausted callback means the job tried and lost; the context is fresh, and you can take action immediately.

Death Handlers: Global Fallback

For jobs that don't have a sidekiq_retries_exhausted callback, you can set a global death handler:

Sidekiq.configure_server do |config|
  config.death_handlers << lambda { |status, exception|
    # Fires when any job dies permanently
    Sentry.capture_exception(exception, extra: {
      job_class: status[:job]['class'],
      jid: status[:job]['jid'],
    })
  }
end

Use this as a safety net for jobs you haven't instrumented individually. In production, both the per-job callback and the global handler can fire for the same job—that's fine; Sentry groups duplicate failures by fingerprint anyway.

Adding Job Context for Actionable Errors

A stack trace without context is nearly useless. "NoMethodError on line 42" doesn't tell you which order, which customer, or what state the system was in. Sidekiq passes job metadata that Sentry can attach automatically—but you can enrich it:

class ProcessPaymentJob
  include Sidekiq::Job
  sidekiq_options retry: 5
  
  def perform(order_id, stripe_customer_id)
    Sentry.set_tags(
      job_class: self.class.name,
      queue: Sidekiq::Queues.get_queue,
    )
    Sentry.set_context('sidekiq', {
      order_id: order_id,
      jid: jid,
      retry_count: Sidekiq::Job::Status.get(jid)&.dig('retry_count'),
    })
    
    order = Order.find(order_id)
    charge_card(stripe_customer_id)
  end
end

Never log raw job arguments if they contain PII or secrets. Use sensitive data scrubbing to redact credit card numbers, API keys, and customer email addresses before they reach LightTrace. The Sentry SDK scrubs common patterns automatically, but verify your scrubbing rules fit your schema.

Idempotency: The Silent Killer

Retries mean at-least-once execution. A job can succeed and then crash before it finishes, and Sidekiq will retry it. If your job isn't idempotent—meaning it can't safely run twice—you've got a problem.

A classic mistake: charging a customer, then crashing before saving the charge to the database. Sidekiq retries, and you charge them twice.

# WRONG: Not idempotent
def perform(customer_id, amount)
  charge_card(customer_id, amount)  # What if this succeeds but we crash?
  Payment.create(customer_id: customer_id, amount: amount)
end

# RIGHT: Idempotent
def perform(payment_id)
  payment = Payment.find_or_create_by(id: payment_id)
  return if payment.charged?  # Already completed
  
  charge_card(payment.customer_id, payment.amount)
  payment.update(charged_at: Time.now, status: 'completed')
end

When designing Sidekiq jobs, always assume they might run twice. Use unique job IDs (payment ID, not customer ID), check for idempotency with find_or_create_by, and only update state after external calls complete. Idempotency isn't optional; it's foundational to reliable background jobs.

Wiring Sidekiq Error Tracking with Sentry

Here's how to set up the Sentry Ruby SDK pointed at LightTrace:

bundle add sentry-ruby sentry-rails sentry-sidekiq

Then initialize in your Sidekiq config:

# config/initializers/sentry.rb
Sentry.init do |config|
  config.dsn = ENV.fetch("SENTRY_DSN", "https://<key>@light-trace.robomiri.com/1")
  config.environment = Rails.env
  config.release = "#{Rails.application.class.module_parent_name}@#{ENV.fetch('APP_VERSION', '0.0.0')}"
  config.traces_sample_rate = 1.0
end

The sentry-sidekiq gem hooks in automatically and captures all job exceptions. Then add per-worker callbacks as needed:

class ProcessPaymentJob
  include Sidekiq::Job
  
  sidekiq_retries_exhausted do |msg, exception|
    Sentry.capture_exception(exception, extra: {
      order_id: msg['args'][0],
      jid: msg['jid'],
    })
  end
  
  def perform(order_id)
    # Job logic here
  end
end

Every fatal job failure lands in LightTrace with the job class, arguments (scrubbed), queue, and retry count attached. You get the full stack trace with breadcrumbs showing what happened before the crash.

Alerting on Job Failures, Not Noise

Alert fatigue kills oncall. You don't need an email for every transient retry. Set alert rules that fire only on real problems:

• New issue: An error type we've never seen before (high priority)
• Frequency threshold: This error has appeared 10 times in the last hour (indicates a systemic problem, not a fluke)
• By tag: Only alert on payment-related jobs, not utility jobs

In LightTrace, create alert rules like "Email me if ProcessPaymentJob fails more than 5 times per hour." That catches cascading failures without alert fatigue. Pair this with error grouping so the same failure—across 100 retries and 10 different job instances—counts as one issue, not thousands.

Also monitor queue health separately:

• Dead letter queue size: If more than 50 jobs are dead, something is broken
• Queue latency: If jobs wait more than 30 seconds to start, workers might be overloaded
• Worker uptime: If workers crash, no errors appear at all

These metrics are complementary to error tracking. One tells you what failed; the other tells you if your background job system is running.

Getting Started

Start with the Sentry gems and the initializer above. Deploy to staging and manually trigger a job exception—verify it appears in LightTrace with full context. Then add sidekiq_retries_exhausted callbacks to your critical jobs. For a deeper dive into error tracking best practices across any framework, see the universal guide to adding error tracking.

If you're coming from a different background job system, Celery in Python or RQ jobs follow the same patterns: capture terminal failures, add context, alert on frequency not retries, and assume at-least-once execution.