Skip to content

Stop processes whose heartbeats keep failing past the alive threshold - #778

Open
wintan1418 wants to merge 1 commit into
rails:mainfrom
wintan1418:heal-pruned-scheduler
Open

Stop processes whose heartbeats keep failing past the alive threshold#778
wintan1418 wants to merge 1 commit into
rails:mainfrom
wintan1418:heal-pruned-scheduler

Conversation

@wintan1418

Copy link
Copy Markdown
Contributor

Related to #763 (and the same family as #751).

What I found

The core scenario in #763 — a process whose registration gets pruned while it's still alive — already self-heals on main: the next heartbeat raises RecordNotFound, Registrable#heartbeat clears the process and wakes the run loop, shutting_down? returns true via !registered?, the fork exits, and the supervisor replaces it. I've added a regression test proving this end-to-end for the scheduler (start a supervisor with a scheduler, delete the scheduler's solid_queue_processes row while it's running, assert a fresh registration appears).

The remaining hole

That self-heal only fires when the heartbeat gets a clean RecordNotFound. Heartbeats can also fail without confirming anything about the registration — a dropped DB connection after host sleep/resume, SQLite busy errors (the reporter's setup in #763) — and those errors are only reported via the timer task's observer. The process then keeps running unregistered indefinitely: the supervisor never replaces it because it never exits, and from the outside it's a zombie that stops doing visible work. This matches the reported evidence: an alive, supervised scheduler with no solid_queue_processes row and no "Replaced terminated Scheduler" ever logged.

Fix

Track when heartbeats started failing. Once they've been failing for longer than SolidQueue.process_alive_threshold — the exact point at which every other process considers this one dead and prunable — treat it like finding the registration gone: clear the process and wake the run loop so the process stops and the supervisor starts a replacement that can register afresh. Errors are still re-raised so the existing observer reporting is unchanged, and a successful heartbeat resets the tracking, so transient blips don't accumulate.

Testing

  • New unit test: a worker whose heartbeats persistently raise ActiveRecord::StatementInvalid shuts down its pool once the failures outlast the alive threshold (fails on main, passes with this change).
  • New regression test for the existing pruned-registration replacement path (passes on main and with this change).
  • Full worker/scheduler/supervisor/process-recovery test files green on MySQL.

A process that finds its registration gone already stops itself so the
supervisor can replace it (RecordNotFound on heartbeat). But heartbeats
can also fail without confirming anything about the registration — a
dropped database connection after the host sleeps and resumes, SQLite
busy errors — and those errors were only reported, leaving the process
running unregistered indefinitely: a zombie that polls or schedules
nothing anyone can see, while the supervisor never replaces it because
it never exits.

Once heartbeats have been failing for longer than the alive threshold,
every other process will have considered this one dead and pruned its
registration anyway, so treat it the same as finding the registration
gone: stop, and let the supervisor start a replacement that can register
afresh.

Also add a regression test covering the existing replacement path for a
scheduler whose registration is pruned while it's still alive.

Related to rails#763
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant