Bound the GPFIFO-space wait in nvWriteGpEntry() - #1192
Conversation
|
|
cf7592c to
2fc027e
Compare
With nvWriteGpEntry() bounded, the same stalled-channel scenario spins in __nvPushMakeRoom() instead, one call earlier on the same path. Bound it with the same NV_PUSH_NOTIFIER_SHORT_TIMEOUT / noTimeout logic. Since this function returns void and callers write methods unconditionally after it returns, on timeout wrap the write pointer back to the buffer base and mark the whole buffer free so the writes stay in bounds; the following kickoff then fails and the caller gives up.
|
Also hitting this on 595.84 with an RTX 5060 Laptop (Blackwell, GB206M,device 10de:2d59). Ubuntu 26.04, GNOME Wayland, custom kernel 7.1.3. After a few hours idle at the lock screen, gnome-shell memory climbs from~500 MB to 700+ MB, DIFR prefetch cycles accumulate, and eventually the shell crashes with a SIGSEGV in the cursor/KMS path. Journal shows the compositor falling back to non-atomic mode setting for nvidia-drm, and the KMS thread blocks. Only a hard reboot recovers. Tried building from this PR's branch, but the GitHub source diverges from Ubuntu's DKMS tarball enough that the resulting modules don't load properly on my system. Would really appreciate this getting merged so it ships in a future driver update. Happy to provide logs if needed |
3 participants
Bound the GPFIFO and pushbuffer waits in nvidia-push
Problem
nvWriteGpEntry() waits for a free GPFIFO entry with no timeout. __nvPushMakeRoom() waits for pushbuffer space with no timeout. Both loops exit only when the GPU makes progress or a channel error is flagged. If a channel stalls without a flagged error, the loop spins forever in kernel context. When the spinning code holds the nvkms lock, every later modeset blocks in D state and the desktop is unusable until reboot.
I hit this reproducibly with DIFR prefetch after resume from suspend. Same trace as #1167 and #1177.
Environment: RTX 5090, driver 595.71.05 open modules, NVreg_PreserveVideoMemoryAllocations enabled, kernel 7.0.3, KDE Plasma on Wayland, S3 suspend.
The nvidia-modeset kthread spins on one core:
and the compositor blocks on the nvkms lock held by that thread:
Fix
Commit 1 bounds nvWriteGpEntry() the same way IdleChannel() already does: poll nvPushImportGetMilliSeconds(), honor the channel noTimeout flag, log an error and return FALSE after NV_PUSH_NOTIFIER_SHORT_TIMEOUT. Kickoff() already handles the FALSE return.
Commit 2 bounds __nvPushMakeRoom() with the same timeout. This function returns void and callers write methods unconditionally after it returns, so on timeout it wraps the write pointer back to the buffer base and marks the whole buffer free. Each method write is asserted smaller than the buffer, so the writes stay in bounds, and they land only in the channel's own pushbuffer. The following kickoff then fails and the caller gives up. Commit 2 is needed because with only commit 1 applied the same stall spins in __nvPushMakeRoom() instead, one call earlier on the same DIFR path. Observed on my machine.
DIFR prefetch already has a failure path for a kickoff that never completes: the semaphore wait in PrefetchSingleSurface() times out and reports NV2080_CTRL_LPWR_DIFR_PREFETCH_FAIL_CE_HW_ERROR. That path was unreachable only because the waits above it could spin first.
Channels created with noTimeout = TRUE keep the old behavior.
Results
Both commits, rebased onto 595.71.05, have been running on my machine since June 25. The stall occurred six times since then, mostly within seconds of resume. Each time the desktop kept working and the driver logged: