fix(replay): Don't let a wedged video encoder freeze the app

[romtsn]

📜 Description

Fixes an ANR where Android apps freeze on foreground/background transitions until the system kills the process.

A reporter pulled the full deobfuscated 131-thread dump from Play Console (the Sentry ANR event only carries main, which is why the lock holder was invisible), giving us the exact chain:

main ──waits──▶ lifecycleLock ──held by──▶ timer thread
                                              │ waits
                                              ▼
                                          encoderLock ──held by──▶ SentryReplayIntegration-0
                                                                       │ parked forever
                                                                       ▼
                                                          MediaCodec.dequeueOutputBuffer

Two cooperating defects, so two bounds — both are needed:

1. SimpleVideoEncoder.drainCodec() had a while (true) with no exit on the endOfStream = true path. Some hardware encoders never emit BUFFER_FLAG_END_OF_STREAM after signalEndOfInputStream(), so the loop spun forever while holding encoderLock. It now gives up after 10 consecutive no-progress iterations (~1s at the existing 100ms poll) and drops the remaining frames.
2. ReplayCache.close() used an unbounded encoderLock.acquire(). It runs inline on the caller's thread from BaseCaptureStrategy.stop(), which ReplayIntegration.stop() invokes while holding lifecycleLock — that nesting is what promotes a stuck encoder into a frozen main thread. It now waits at most 2s and skips the release otherwise.

The loop bound alone isn't sufficient: the native frames show MediaCodec::dequeueOutputBuffer → AMessage::postAndAwaitResponse → ALooper::awaitResponse with MediaCodec_looper idle. TIMEOUT_USEC is handed to the codec looper, which is supposed to reply once it elapses — a dead looper never replies, and awaitResponse has no deadline of its own. So a single call can never return, and only bounding the lock wait keeps the lifecycle path free.

On the timeout path the (already-dead) codec is not released, leaking a native handle. That's the deliberate trade: leaking a handle in a process whose encoder is wedged beats freezing the app. isClosed.set(true) still runs on every path, since persistSegmentValues gates on it.

The other four encoderLock.acquire() sites in ReplayCache are unchanged — they all run on the replay worker, the thread that would be stuck, so a deadline there buys nothing.

Adds AutoClosableReentrantLock.tryAcquire(timeout, unit) (@ApiStatus.Internal, additive) for the second bound. Returns the token on success and null on timeout, so callers must branch explicitly rather than silently no-op via ?.use {}.

Net effect: a wedged codec degrades to "replay stops working for this process" instead of "the app freezes."

💚 How did you test it?

ReplayShadowMediaCodec gained two hooks, since it ignores timeoutUs and always eventually produces EOS: neverSignalEos and blockOnDequeue. New tests cover both bounds in ReplayCacheTest, plus 4 for tryAcquire in AutoClosableReentrantLockTest.

Both ReplayCacheTest cases were mutation-checked — reverting each fix in turn makes the matching test fail with its assertion message rather than hang. Full :sentry-android-replay:testReleaseUnitTest passes; apiDump adds exactly the one tryAcquire line.

No device repro is available (Redmi Note 14 Pro 5G / Pro+ 5G, Android 16, MediaTek), so the shadow-based tests are the verification of record.

📝 Checklist

• I reviewed the submitted code.
• I added tests to verify the changes.
• No new PII added or SDK only sends newly added PII if sendDefaultPii is enabled.
• I updated the docs if needed.
• No breaking change or entry added to the changelog.

🤖 Generated with Claude Code

[runningcode]

how did we determine 10 here? Just curious. How fast do these iterations happen?

[sentry]

📲 Install Builds

Android

🔗 App Name	App ID	Version	Configuration
SDK Size	io.sentry.tests.size	8.50.0 (1)	release

⚙️ sentry-android Build Distribution Settings

[github-actions]

Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	373.74 ms	470.27 ms	96.53 ms
Size	0 B	0 B	0 B

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
05aa61d	326.06 ms	385.46 ms	59.40 ms
bb0ff41	315.84 ms	350.76 ms	34.92 ms
806307f	357.85 ms	424.64 ms	66.79 ms
d501a7e	307.33 ms	341.94 ms	34.61 ms
0ee65e9	321.06 ms	361.24 ms	40.18 ms
ed33deb	334.19 ms	362.30 ms	28.11 ms
9fbb112	401.87 ms	515.87 ms	114.00 ms
b8bd880	314.56 ms	336.50 ms	21.94 ms
5b1a06b	315.40 ms	353.33 ms	37.94 ms
6edfca2	316.43 ms	398.90 ms	82.46 ms

App size

Revision	Plain	With Sentry	Diff
05aa61d	0 B	0 B	0 B
bb0ff41	0 B	0 B	0 B
806307f	1.58 MiB	2.10 MiB	533.42 KiB
d501a7e	0 B	0 B	0 B
0ee65e9	0 B	0 B	0 B
ed33deb	1.58 MiB	2.13 MiB	559.52 KiB
9fbb112	1.58 MiB	2.11 MiB	539.18 KiB
b8bd880	1.58 MiB	2.29 MiB	722.92 KiB
5b1a06b	0 B	0 B	0 B
6edfca2	1.58 MiB	2.13 MiB	559.07 KiB

[runningcode]

looks good, i left a question for a discussion!

[runningcode]

do we need to surround this with try/catch as well?

[runningcode]

ok so just to check my understanding, we try to get the lock only to make sure nobody else is holding it before calling release but if we can't acquire the lock, what state are we in? is this safe to skip the release? won't we cause a memory leak?