Skip to content

Replace terminated forks even if releasing their claimed jobs fails - #781

Open
wintan1418 wants to merge 1 commit into
rails:mainfrom
wintan1418:replace-terminated-forks-when-db-is-down
Open

Replace terminated forks even if releasing their claimed jobs fails#781
wintan1418 wants to merge 1 commit into
rails:mainfrom
wintan1418:replace-terminated-forks-when-db-is-down

Conversation

@wintan1418

Copy link
Copy Markdown
Contributor

Fixes #683

Problem

When the database restarts under a running fork supervisor, the workers and dispatcher crash and exit (their polling loops raise). The supervisor reaps them and calls replace_fork, which runs release_claimed_jobs_by — a database call — before starting the replacement. With the database still down, that call raises, the exception escapes check_and_replace_terminated_processes, and the whole supervise loop crashes without ever replacing the terminated forks.

The scheduler, whose per-task enqueue errors are reported without killing the process, survives and reconnects — so after the database comes back, recurring jobs keep being enqueued with no workers left to run them. I reproduced this end-to-end on main with docker compose restart mysql under bin/jobs: a growing backlog of ready executions, worker heartbeats frozen at the restart timestamp, and a supervisor that never recovers. Full chain of evidence — including the recovered crash backtrace and why the process hangs instead of exiting when the debug gem is loaded — is written up in a comment on #683.

Fix

Rescue errors from release_claimed_jobs_by in replace_fork, report them through handle_thread_error, and start the replacement regardless. The database being unreachable at that moment is likely the very reason the fork died, so the replacement can't depend on it.

The terminated fork's claimed jobs aren't lost by skipping the release: its stale process registration is pruned once the database is reachable again, and pruning fails its claimed executions through the normal ProcessPrunedError path.

Testing

  • New unit test: replace_fork still starts the replacement when releasing claimed jobs raises ConnectionNotEstablished (errors without this change, passes with it).
  • Live verification of the original scenario: with this change, bin/jobs under a docker compose restart mysql recovers by itself — replacement workers crash-loop through the boot-timeout path while the database is down, then re-register and drain the backlog as soon as it's back. On main, the same scenario ends with a permanently dead supervisor loop and an ever-growing backlog.

When the database restarts under a running fork supervisor, the workers
and dispatcher crash and exit. The supervisor reaps them and calls
replace_fork, which released the terminated fork's claimed jobs — a
database call — before starting the replacement. With the database still
down that call raised, the exception crashed the supervise loop, and the
terminated forks were never replaced: the scheduler (which survives
enqueue errors) kept enqueueing recurring jobs that nothing would ever
process, until the whole thing was restarted by hand.

Rescue errors from the release, report them through handle_thread_error
and start the replacement regardless. The database being unreachable at
that point is likely the very reason the fork died, so replacing it
can't depend on the database being back. The terminated fork's claimed
jobs aren't lost: its stale registration gets pruned once the database
is reachable again, which fails them through the regular path so they
can be retried.

Fixes rails#683
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SolidQueue stops processing any jobs, but does not actually exit, when DB restarts unless --skip-recurring is set

1 participant