Stop processes whose heartbeats keep failing past the alive threshold - #778
Open
wintan1418 wants to merge 1 commit into
Open
Stop processes whose heartbeats keep failing past the alive threshold#778wintan1418 wants to merge 1 commit into
wintan1418 wants to merge 1 commit into
Conversation
A process that finds its registration gone already stops itself so the supervisor can replace it (RecordNotFound on heartbeat). But heartbeats can also fail without confirming anything about the registration — a dropped database connection after the host sleeps and resumes, SQLite busy errors — and those errors were only reported, leaving the process running unregistered indefinitely: a zombie that polls or schedules nothing anyone can see, while the supervisor never replaces it because it never exits. Once heartbeats have been failing for longer than the alive threshold, every other process will have considered this one dead and pruned its registration anyway, so treat it the same as finding the registration gone: stop, and let the supervisor start a replacement that can register afresh. Also add a regression test covering the existing replacement path for a scheduler whose registration is pruned while it's still alive. Related to rails#763
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #763 (and the same family as #751).
What I found
The core scenario in #763 — a process whose registration gets pruned while it's still alive — already self-heals on
main: the next heartbeat raisesRecordNotFound,Registrable#heartbeatclears the process and wakes the run loop,shutting_down?returns true via!registered?, the fork exits, and the supervisor replaces it. I've added a regression test proving this end-to-end for the scheduler (start a supervisor with a scheduler, delete the scheduler'ssolid_queue_processesrow while it's running, assert a fresh registration appears).The remaining hole
That self-heal only fires when the heartbeat gets a clean
RecordNotFound. Heartbeats can also fail without confirming anything about the registration — a dropped DB connection after host sleep/resume, SQLite busy errors (the reporter's setup in #763) — and those errors are only reported via the timer task's observer. The process then keeps running unregistered indefinitely: the supervisor never replaces it because it never exits, and from the outside it's a zombie that stops doing visible work. This matches the reported evidence: an alive, supervised scheduler with nosolid_queue_processesrow and no "Replaced terminated Scheduler" ever logged.Fix
Track when heartbeats started failing. Once they've been failing for longer than
SolidQueue.process_alive_threshold— the exact point at which every other process considers this one dead and prunable — treat it like finding the registration gone: clear the process and wake the run loop so the process stops and the supervisor starts a replacement that can register afresh. Errors are still re-raised so the existing observer reporting is unchanged, and a successful heartbeat resets the tracking, so transient blips don't accumulate.Testing
ActiveRecord::StatementInvalidshuts down its pool once the failures outlast the alive threshold (fails onmain, passes with this change).mainand with this change).