Skip to content

Roll back transactions leaked by killed job threads in tests - #773

Merged
rosa merged 1 commit into
mainfrom
rollback-transactions-leaked-by-killed-job-threads
Jul 30, 2026
Merged

Roll back transactions leaked by killed job threads in tests#773
rosa merged 1 commit into
mainfrom
rollback-transactions-leaked-by-killed-job-threads

Conversation

@rosa

@rosa rosa commented Jul 30, 2026

Copy link
Copy Markdown
Member

Since #766, tests that stop in-process workers kill the job threads those workers leak. That kill can interrupt a thread in the middle of a database write, skipping the rollback in the transaction's ensure block while still returning the connection to the pool — with the write transaction still open.

On SQLite that open transaction locks out every other writer. Nothing clears it until the connection pool's reaper flushes the idle connection at its default 300-second idle_timeout, so a single unlucky kill turns into a ~5-minute cascade of SQLite3::BusyException: database is locked errors through whatever tests run next, after which the suite mysteriously heals.

That's been the signature of the recent flaky sqlite legs on CI: dozens of errors, always starting with SolidQueue::JobTest#test_try_to_discard_claimed_job failing in its own teardown ~10 seconds after killing a job thread (it's the only non-transactional caller of the helper), always healing ~300 seconds later. For example, this run and this one, both on main.

The fix: after killing the threads, wait for them to die, then drop all pool connections, which rolls back anything they left open. Transactional tests don't need it — they pin every thread to the test's own connection, so a killed thread can't strand a lock on a connection of its own.

The leaked state and the fix are easy to see in isolation: a thread killed while holding an open transaction leaves the pool with raw_connection.transaction_active? == true and the next writer times out; after disconnect!, the write goes through instantly.

🤖 Generated with Claude Code

Killing the job threads leaked by in-process workers can interrupt a
thread in the middle of a database write, skipping the rollback in the
transaction's ensure block while still returning the connection to the
pool. The open transaction then locks SQLite for every other writer
until the pool reaper flushes the idle connection minutes later,
cascading "database is locked" errors through the rest of the suite.

Wait for the killed threads to die and drop all pool connections,
rolling back anything they left open. Transactional tests don't need
it: they pin every thread to the test's own connection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rosa
rosa merged commit 7f59932 into main Jul 30, 2026
172 of 174 checks passed
@rosa
rosa deleted the rollback-transactions-leaked-by-killed-job-threads branch July 30, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant