From 37f1265896845c582181d7c29a499802048a0eec Mon Sep 17 00:00:00 2001
From: wintan1418 <wintan1418@gmail.com>
Date: Thu, 30 Jul 2026 14:14:25 +0100
Subject: [PATCH] Include the error in fail_many_claimed events and document
 recovery

Jobs running on a process that dies without a clean shutdown are marked
as failed with ProcessPrunedError (or ProcessExitError when the
supervisor reaps a crashed fork) via ClaimedExecution.fail_all_with,
which already emits a fail_many_claimed event with the affected job and
process ids. The error itself wasn't part of the payload though, so
subscribers couldn't tell these mass failures apart from each other.

Add the error to the event payload and to the log line, and document how
to subscribe to the event to implement a custom recovery policy for
these jobs, including why Solid Queue doesn't retry them automatically.

Closes rails/solid_queue#591
---
 README.md                                   | 22 +++++++++++++++++++++
 app/models/solid_queue/claimed_execution.rb |  1 +
 lib/solid_queue/log_subscriber.rb           |  5 ++++-
 test/integration/instrumentation_test.rb    |  3 +++
 4 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index a0b9a2ea..ef10ed8d 100644
--- a/README.md
+++ b/README.md
@@ -30,6 +30,7 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite,
   - [Performance considerations](#performance-considerations)
 - [Failed jobs and retries](#failed-jobs-and-retries)
   - [Error reporting on jobs](#error-reporting-on-jobs)
+  - [Jobs interrupted by non-graceful process death](#jobs-interrupted-by-non-graceful-process-death)
 - [Puma plugin](#puma-plugin)
 - [Jobs and transactional integrity](#jobs-and-transactional-integrity)
 - [Recurring tasks](#recurring-tasks)
@@ -664,6 +665,27 @@ class ApplicationMailer < ActionMailer::Base
 end
 ```
 
+### Jobs interrupted by non-graceful process death
+
+When a process dies without a clean shutdown (for example, `SIGKILL`ed by the OS or the container runtime because of memory limits), the jobs it was running can't be released back to their queues. Once another process notices the missing heartbeats and prunes the dead process's registration, its in-flight jobs are marked as failed with `SolidQueue::Processes::ProcessPrunedError`. Solid Queue deliberately doesn't retry these automatically: the job itself might be what's killing the process (for example, a job that exhausts the container's memory), and retrying it blindly would just kill the next worker too.
+
+If you know your jobs are idempotent and want to implement your own recovery policy, you can subscribe to the `fail_many_claimed.solid_queue` event, which includes the error and the affected job IDs in its payload:
+
+```ruby
+# config/initializers/solid_queue_recovery.rb
+ActiveSupport::Notifications.subscribe("fail_many_claimed.solid_queue") do |event|
+  if event.payload[:error].is_a?(SolidQueue::Processes::ProcessPrunedError)
+    SolidQueue::FailedExecution.where(job_id: event.payload[:job_ids]).each do |failed_execution|
+      # Apply your own safeguard against retrying the same job in a loop,
+      # e.g. a counter in the job's arguments or a cap stored elsewhere.
+      failed_execution.retry
+    end
+  end
+end
+```
+
+The event is emitted in the process that performs the pruning (or the supervisor when it reaps a crashed fork, with `SolidQueue::Processes::ProcessExitError`), so make sure the subscription is set up in an initializer, where all Solid Queue processes will load it.
+
 ## Puma plugin
 
 We provide a Puma plugin if you want to run the Solid Queue's supervisor together with Puma and have Puma monitor and manage it. You just need to add
diff --git a/app/models/solid_queue/claimed_execution.rb b/app/models/solid_queue/claimed_execution.rb
index 26569269..9a606e0b 100644
--- a/app/models/solid_queue/claimed_execution.rb
+++ b/app/models/solid_queue/claimed_execution.rb
@@ -48,6 +48,7 @@ def fail_all_with(error)
           payload[:process_ids] = executions.map(&:process_id).uniq
           payload[:job_ids] = executions.map(&:job_id).uniq
           payload[:size] = executions.size
+          payload[:error] = error
         end
       end
     end
diff --git a/lib/solid_queue/log_subscriber.rb b/lib/solid_queue/log_subscriber.rb
index bb6b67da..4bb52baf 100644
--- a/lib/solid_queue/log_subscriber.rb
+++ b/lib/solid_queue/log_subscriber.rb
@@ -16,7 +16,10 @@ def release_many_claimed(event)
   end
 
   def fail_many_claimed(event)
-    warn formatted_event(event, action: "Fail claimed jobs", **event.payload.slice(:job_ids, :process_ids))
+    attributes = event.payload.slice(:job_ids, :process_ids)
+    attributes[:error] = formatted_error(event.payload[:error]) if event.payload[:error]
+
+    warn formatted_event(event, action: "Fail claimed jobs", **attributes)
   end
 
   def release_claimed(event)
diff --git a/test/integration/instrumentation_test.rb b/test/integration/instrumentation_test.rb
index 3ff0b13f..4fd8a295 100644
--- a/test/integration/instrumentation_test.rb
+++ b/test/integration/instrumentation_test.rb
@@ -164,6 +164,9 @@ class InstrumentationTest < ActiveSupport::TestCase
 
     assert_equal 1, events.count
     assert_event events.first, "fail_many_claimed", process_ids: [ process.id ], job_ids: jobs.map(&:id), size: 3
+
+    error = events.first.last[:error]
+    assert_instance_of SolidQueue::Processes::ProcessPrunedError, error
   end
 
   test "errors when deregistering processes are included in deregister_process events" do