Give in-flight jobs enough pause to survive slow shutdowns in tests - #782
Give in-flight jobs enough pause to survive slow shutdowns in tests#782wintan1418 wants to merge 3 commits into
Conversation
Three tests assert that a job paused mid-flight does NOT complete while its process is terminated: the async and forked "term supervisor exceeding timeout" tests and "don't block claimed executions that get released". They paused the job for shutdown_timeout + 10 seconds, but that clock starts when the job starts executing, and on a slow CI runner the test's detection waits (up to 8 seconds of polling for the job to show as started) plus the termination window (up to 7 more) can outlast it: the sleep finishes inside the still-alive process, the job writes its completed status, and the assertion fails. Widen the pause to shutdown_timeout + 30 seconds. The job is killed mid-sleep either way, so the test's runtime doesn't change; only the margin against slow runners does. Related to rails#602
The pause job's sleep is far longer than the test's entire timeline, so when this assertion fails on CI with the job completed, the write must have come from outside the test's own flow — a process or thread leaked from a neighboring test. That can't be diagnosed from "expected completed to not be equal to completed", so include all job results and the registered processes in the failure message.
|
Update: the async variant of this flake fired again on this PR's own CI run, so the widened pause alone doesn't fully explain it. Notably, with the pause now at |
The test waits for the job's "started" result before SIGKILLing the worker, but with the non-raising wait and only 2 seconds of margin: on a slow runner, worker boot plus polling plus claiming can take longer, the guard gives up silently, and the KILL lands before the job ever starts — so the "started" row the test asserts on later never gets written. Use the raising variant with a 10 second margin, so a job that never starts fails the test right at the guard, and widen the job's pause so the KILL can't arrive after the pause has already finished on a slow runner (which would complete the job and erase the "started" status too). Neither change affects the test's runtime in the intended flow.
|
Pushed one more fix in the same family: |
Related to #602. Companion to #776, targeting the next-most-frequent flakes in recent CI runs:
AsyncProcessesLifecycleTest#test_term_supervisor_exceeding_timeout_while_there_are_jobs_in-flight(and its forked twin) andConcurrencyControlsTest#test_don't_block_claimed_executions_that_get_released, each seen failing standalone on recentmain/PR runs with:The race
All three tests enqueue a job that pauses for
shutdown_timeout + 10.secondsand assert it does not complete while its process is terminated mid-flight. But the pause clock starts when the job begins executing, not when the test detects it started. On a slow runner, the detection polling (up to ~8s) plus the termination window (up to ~7s) can add up to more than the pause: thesleepfinishes inside the still-alive process, the job writescompleted, and the assertion fails. Nothing kills the sleeping thread earlier —wait_for_terminationonly waits, so the job thread survives until actual process death.Fix
Widen the pause to
shutdown_timeout + 30.seconds. Since the job is killed mid-sleep in every intended outcome, this doesn't change any test's runtime — only the margin against slow runners (from ~2s of slack to ~22s).Verified all three tests green on MySQL locally; runtime unchanged (the two lifecycle tests together still run in ~6s).