Reference chains for surviving live-heap samples (PROF-15341)#644
Draft
jbachorik wants to merge 20 commits into
Draft
Reference chains for surviving live-heap samples (PROF-15341)#644jbachorik wants to merge 20 commits into
jbachorik wants to merge 20 commits into
Conversation
Contributor
|
Contributor
CI Test ResultsRun: #29577544496 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-17 11:56:50 UTC |
Contributor
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/125072101 |
Contributor
Benchmark Results (commit cfec595)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1949 / 2055 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8727 / 8389 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8166 / 8396 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1292 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1288 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2830 / 3023 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2976 / 2821 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | · / ✅ | · / ✅ | · / 3 | · / 3555 | · / ✅ | · / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3499 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1592 / 1697 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1922 / 1854 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 10, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 0714bd3)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10184 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +1.8% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8891 ms (24 iters) | ✅ 8865 ms (24 iters) | ≈ -0.3% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6003 ms (33 iters) | ✅ 6034 ms (33 iters) | ≈ +0.5% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5507 ms (36 iters) | ✅ 5426 ms (36 iters) | ≈ -1.5% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2834 ms (66 iters) | ✅ 2761 ms (67 iters) | 🟢 -2.6% | — / — |
| fj-kmeans | 25 | ✅ 2824 ms (66 iters) | ✅ 2838 ms (66 iters) | ≈ +0.5% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2113 ms (88 iters) | ✅ 2065 ms (89 iters) | ≈ -2.3% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2059 ms (90 iters) | ✅ 2102 ms (89 iters) | ≈ +2.1% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1283 ms (134 iters) | ✅ 1260 ms (135 iters) | ≈ -1.8% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 16658 ms (15 iters) | ✅ 16371 ms (16 iters) | ≈ -1.7% (±9%) | — / — |
| reactors | 25 | ✅ 18507 ms (15 iters) | ✅ 18056 ms (15 iters) | ≈ -2.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1929 / 2034 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2310 / 2113 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 8529 / 8852 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8635 / 8390 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 1284 / 1281 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 1283 / 1285 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2930 / 2864 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2963 / 2953 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3558 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3463 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1564 / 1782 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1681 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 700d838)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275960 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10425 ms (21 iters) | ✅ 10259 ms (21 iters) | ≈ -1.6% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8748 ms (24 iters) | ✅ 8797 ms (24 iters) | ≈ +0.6% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5970 ms (33 iters) | ✅ 5982 ms (33 iters) | ≈ +0.2% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5468 ms (36 iters) | ≈ -0% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2598 ms (73 iters) | ✅ 2715 ms (69 iters) | 🔴 +4.5% | — / — |
| fj-kmeans | 25 | ✅ 2784 ms (67 iters) | ✅ 2847 ms (66 iters) | ≈ +2.3% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2122 ms (87 iters) | ✅ 2126 ms (88 iters) | ≈ +0.2% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2035 ms (91 iters) | ✅ 2057 ms (91 iters) | ≈ +1.1% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1304 ms (131 iters) | ✅ 1253 ms (136 iters) | ≈ -3.9% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 1018 ms (168 iters) | ≈ +0.3% (±31.8%) | — / — |
| reactors | 21 | ✅ 16107 ms (15 iters) | ✅ 16847 ms (15 iters) | ≈ +4.6% (±7.9%) | — / — |
| reactors | 25 | ✅ 18469 ms (15 iters) | ✅ 19075 ms (14 iters) | ≈ +3.3% (±4.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2228 / 2453 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8686 / 8417 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8561 / 8290 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1272 / 1289 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1287 / 1298 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2912 / 3017 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2907 / 2872 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3515 / 3489 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 7 | 3498 / 3463 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1732 / 1701 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1780 / 1924 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 14, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 14, 2026 12:03
700d838 to
9972bff
Compare
Contributor
Benchmark Results (commit 9972bff)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124511782 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a636398)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124700626 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit f2d8978)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124747648 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a48899b)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124773626 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10376 ms (21 iters) | ✅ 10228 ms (21 iters) | ≈ -1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8829 ms (24 iters) | ✅ 8859 ms (24 iters) | ≈ +0.3% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 6001 ms (33 iters) | ✅ 5928 ms (33 iters) | ≈ -1.2% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5475 ms (36 iters) | ≈ +0.1% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2687 ms (70 iters) | ✅ 2761 ms (68 iters) | 🔴 +2.8% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2805 ms (66 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2047 ms (91 iters) | ✅ 2116 ms (88 iters) | 🔴 +3.4% | — / — |
| future-genetic | 25 | ✅ 2000 ms (93 iters) | ✅ 2083 ms (89 iters) | 🔴 +4.2% | — / — |
| naive-bayes | 21 | ✅ 1256 ms (136 iters) | ✅ 1253 ms (136 iters) | ≈ -0.2% (±32.8%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1016 ms (169 iters) | ≈ 0% (±31.3%) | — / — |
| reactors | 21 | ✅ 16870 ms (15 iters) | ✅ 16463 ms (15 iters) | ≈ -2.4% (±7.3%) | — / — |
| reactors | 25 | ✅ 18270 ms (15 iters) | ✅ 18333 ms (15 iters) | ≈ +0.3% (±4.5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1988 / 2024 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2263 / 2395 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 6 | 8440 / 8679 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8283 / 8410 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1288 / 1260 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 1250 / 1295 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2968 / 3003 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2843 / 2902 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3474 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3495 / 3488 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1726 / 1721 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1893 / 1750 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 787f29e)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124811024 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10215 ms (21 iters) | ✅ 10475 ms (21 iters) | ≈ +2.5% (±11.9%) | — / — |
| akka-uct | 25 | ✅ 8843 ms (24 iters) | ✅ 8861 ms (24 iters) | ≈ +0.2% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ -0.3% (±25.5%) | |
| finagle-chirper | 25 | ✅ 5515 ms (36 iters) | ✅ 5479 ms (36 iters) | ≈ -0.7% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2693 ms (70 iters) | ✅ 2836 ms (66 iters) | 🔴 +5.3% | — / — |
| fj-kmeans | 25 | ✅ 2820 ms (66 iters) | ✅ 2837 ms (66 iters) | ≈ +0.6% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2059 ms (90 iters) | ✅ 2070 ms (89 iters) | ≈ +0.5% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2089 ms (89 iters) | ✅ 2130 ms (87 iters) | ≈ +2% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1284 ms (133 iters) | ✅ 1313 ms (131 iters) | ≈ +2.3% (±33%) | — / — |
| naive-bayes | 25 | ✅ 1009 ms (170 iters) | ✅ 988 ms (173 iters) | ≈ -2.1% (±31.4%) | — / — |
| reactors | 21 | ✅ 16540 ms (15 iters) | ✅ 16155 ms (15 iters) | ≈ -2.3% (±7.1%) | — / — |
| reactors | 25 | ✅ 18545 ms (15 iters) | ✅ 18816 ms (15 iters) | ≈ +1.5% (±4.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1987 / 1990 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2238 / 2323 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8290 / 8529 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8323 / 8386 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1299 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1274 / 1299 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2920 / 2982 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2886 / 2864 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3505 / 3499 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1593 / 1669 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1849 / 1940 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit fede524)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124833099 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit ef0f5c6)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124961539 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 1299316)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124967829 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10336 ms (21 iters) | ✅ 10245 ms (21 iters) | ≈ -0.9% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8868 ms (24 iters) | ✅ 8789 ms (24 iters) | ≈ -0.9% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 5968 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ +0.1% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5510 ms (36 iters) | ✅ 5537 ms (36 iters) | ≈ +0.5% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2696 ms (69 iters) | ✅ 2822 ms (66 iters) | 🔴 +4.7% | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2746 ms (68 iters) | 🟢 -3.4% | — / — |
| future-genetic | 21 | ✅ 2068 ms (90 iters) | ✅ 2051 ms (90 iters) | ≈ -0.8% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2134 ms (87 iters) | ✅ 2059 ms (90 iters) | 🟢 -3.5% | — / — |
| naive-bayes | 21 | ✅ 1318 ms (130 iters) | ✅ 1253 ms (136 iters) | ≈ -4.9% (±31.8%) | — / — |
| naive-bayes | 25 | ✅ 998 ms (171 iters) | ✅ 1025 ms (167 iters) | ≈ +2.7% (±32.1%) | — / — |
| reactors | 21 | ✅ 16264 ms (15 iters) | ✅ 16625 ms (15 iters) | ≈ +2.2% (±7.4%) | — / — |
| reactors | 25 | ✅ 17926 ms (15 iters) | ✅ 18917 ms (15 iters) | ≈ +5.5% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1917 / 1934 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 2285 / 2434 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8503 / 8628 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8331 / 8664 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1287 / 1266 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1277 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 3037 / 2981 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2897 / 2943 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 12 | 3507 / 3493 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 5 | 3446 / 3481 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1765 / 1665 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1855 / 1920 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 16, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 16, 2026 11:31
1299316 to
baba070
Compare
Contributor
Benchmark Results (commit baba070)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124983919 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 35c49c2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997895 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10367 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +0% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8917 ms (24 iters) | ✅ 8766 ms (24 iters) | ≈ -1.7% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5972 ms (33 iters) | ≈ -0.2% (±25.3%) | |
| fj-kmeans | 21 | ✅ 2801 ms (67 iters) | ✅ 2750 ms (68 iters) | ≈ -1.8% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2756 ms (68 iters) | 🟢 -3% | — / — |
| future-genetic | 21 | ✅ 2142 ms (87 iters) | ✅ 2080 ms (89 iters) | 🟢 -2.9% | — / — |
| future-genetic | 25 | ✅ 1991 ms (94 iters) | ✅ 2035 ms (91 iters) | ≈ +2.2% (±2.7%) | — / — |
| naive-bayes | 25 | ✅ 1025 ms (167 iters) | ✅ 1010 ms (169 iters) | ≈ -1.5% (±31.4%) | — / — |
| reactors | 21 | ✅ 16244 ms (15 iters) | ✅ 16474 ms (15 iters) | ≈ +1.4% (±7.9%) | — / — |
| reactors | 25 | ✅ 18288 ms (15 iters) | ✅ 18332 ms (15 iters) | ≈ +0.2% (±3.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2303 / 2180 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 8767 / 8881 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8034 / 8300 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1316 / 1287 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1289 / 1282 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2889 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2935 / 2838 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3496 / 3495 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3477 / 3510 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1642 / 1586 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1802 / 1759 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit a9fb05f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125008807 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10453 ms (21 iters) | ≈ +1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8814 ms (24 iters) | ✅ 8779 ms (24 iters) | ≈ -0.4% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 6006 ms (33 iters) | ✅ 6047 ms (33 iters) | ≈ +0.7% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5467 ms (36 iters) | ✅ 5455 ms (36 iters) | ≈ -0.2% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2772 ms (68 iters) | ✅ 2687 ms (70 iters) | 🟢 -3.1% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2801 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2087 ms (88 iters) | ✅ 2055 ms (90 iters) | ≈ -1.5% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2088 ms (89 iters) | ✅ 2046 ms (91 iters) | ≈ -2% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1233 ms (139 iters) | ✅ 1257 ms (136 iters) | ≈ +1.9% (±32.9%) | — / — |
| naive-bayes | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
| reactors | 21 | ✅ 16947 ms (15 iters) | ✅ 15936 ms (15 iters) | ≈ -6% (±8.2%) | — / — |
| reactors | 25 | ✅ 18681 ms (15 iters) | ✅ 18731 ms (15 iters) | ≈ +0.3% (±5.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2008 / 2140 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2058 / 2262 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8437 / 8783 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8064 / 8228 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 6 / ✅ | 1275 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1250 / 1261 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2876 / 3036 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2912 / 2869 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3492 / 3491 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1892 / 1633 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1914 / 1937 | ✅ / ✅ | ✅ / ✅ |
3 tasks
Contributor
Benchmark Results (commit 2033006)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125072159 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10363 ms (21 iters) | ✅ 10147 ms (21 iters) | ≈ -2.1% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8966 ms (24 iters) | ✅ 8873 ms (24 iters) | ≈ -1% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6007 ms (33 iters) | ✅ 5986 ms (33 iters) | ≈ -0.3% (±26.1%) | |
| finagle-chirper | 25 | ✅ 5488 ms (36 iters) | ✅ 5433 ms (36 iters) | ≈ -1% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2726 ms (68 iters) | ✅ 2760 ms (68 iters) | ≈ +1.2% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2848 ms (66 iters) | ✅ 2743 ms (69 iters) | 🟢 -3.7% | — / — |
| future-genetic | 21 | ✅ 2062 ms (90 iters) | ✅ 2047 ms (90 iters) | ≈ -0.7% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1994 ms (93 iters) | ✅ 2057 ms (90 iters) | 🔴 +3.2% | — / — |
| naive-bayes | 21 | ✅ 1269 ms (135 iters) | ✅ 1258 ms (136 iters) | ≈ -0.9% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1016 ms (168 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16554 ms (15 iters) | ✅ 16630 ms (15 iters) | ≈ +0.5% (±8.5%) | — / — |
| reactors | 25 | ✅ 18407 ms (15 iters) | ✅ 18308 ms (15 iters) | ≈ -0.5% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2101 / 2026 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 2336 / 2365 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8772 / 8497 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8397 / 8470 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1260 / 1274 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1303 / 1307 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2893 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2958 / 2912 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3540 / 3527 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3458 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1736 / 1754 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1990 / 1872 | ✅ / ✅ | ✅ / ✅ |
GC-root reconstruction for live-heap allocation samples: track surviving objects, walk the reference graph back to a GC root, and emit the resulting chain via JFR. Core support spans the liveness tracker, object sampler, flight recorder, and JVMTI-facing safe-memory-access helpers used while walking potentially-stale references.
Gtest coverage for the liveness tracker, reference-chain BFS reconstruction, JFR round-trip encoding, and the line-number-table copy bounds guard (catches an off-by-one mutation at the MAX_LINE_NUMBER_TABLE_ENTRIES boundary).
In-process and external-process end-to-end coverage: JFR event parsing, a leaking-cache scenario that grows a live population over time, and test seams that isolate frontier/BFS state across shared-JVM test runs. Also fixes a stale-jmethodID stress test's javadoc to correctly attribute which SafeAccess guard rejects which access.
Wire reference-chain tests into the test workflow and fix quoting/allocator existence-check gaps in run-chaos-harness.sh surfaced by review.
Contributor
Bits has a CI fix ready🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready
View in Datadog | Reviewed commit 6f17532 · Any feedback? Reach out in #deveng-pr-agent |
Adds a rootKind string field to datadog.ReferenceChain, labeling why a chain is reachable at all (JNI global, thread stack, static field, ...) rather than just the referrer-class hops already in the chain array. FrontierEntry gains a root_kind byte (the jvmtiHeapReferenceKind of the admitting edge, set only on root-attached entries), threaded through FrontierTable::reconstructChain()'s new out-param into ReferenceChainEvent::_root_kind and written out via a small rootKindName() lookup in Recording::recordReferenceChain(). Chose a labeled scalar field over a synthetic node in the chain array itself, since that array is a JFR T_CLASS cpool array with no room for a non-class placeholder. Extends the JFR roundtrip test to exercise the new field end to end. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…cts design doc Adds native-allocation counters (REFERENCE_CHAIN_FRONTIER_TABLE_BYTES/_CAPACITY) to attribute FrontierTable off-heap RSS that bypasses os::malloc and is therefore invisible to NMT. Adds a repro comparison script and repro-harness tweaks used to measure it. Adds a design doc proposing a VMStructs-based traversal alternative to JVMTI's FollowReferences, to fix the permanent root-subgraph-miss risk from HotSpot's unspecified DFS traversal order. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…Structs design Adds the adversarial-review findings (safepoint/callback constraints, Shenandoah/ZGC barrier-safety gap, stack-root resumability gap) plus their fixes: gate the manual-walk path on G1 only, and route stack-root discoveries through the existing tag-indexed frontier table instead of persisting root descriptors. Adds a source-verified addendum on Shenandoah (mark-word forwarding, tractable as a v1.5 follow-up) vs ZGC (colored-pointer decode is runtime-computed and phase-dependent, not just version-fragile, so it stays excluded). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Corrects the G1-only gating justification (it is SafepointSynchronize plus SuspendibleThreadSet stopping G1's concurrent JavaThread workers, not VM_Operation serialization - conclusion holds, mechanism was wrong) and flags a pre-existing root-attribution staleness bug the stack-root fix surfaced but does not solve: root_kind is captured once at first discovery and never refreshed, so a transient stack-local (or JNI_LOCAL) root can be reported as the reason an object is alive long after that root is gone. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fold in findings from prior-art research (MAT, JFR, dotMemory, LeakCanary) on the pre-existing bug where root_kind is captured once at first admission and never refreshed. Propose a three-part fix: durable-root tie-break at admission time, honest "first observed via" labeling in output, and opportunistic upgrade via the already-cheap per-pass root re-enumeration. Explicitly flag the residual gap (reachability restored via an intermediate object, not directly from a root) as an accepted limitation rather than solving it. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Address the previously-accepted limitation (reachability restored via an intermediate already-tracked object, not directly from a root) by feeding a small, budgeted, rotating subset of already-EXPANDED entries back into the existing frontier/budget queue each pass, prioritized by transient root_kind. Reuses the existing admission callback and budget mechanism instead of introducing a second one; turns "never re-verified" into "re-verified within a bounded number of passes." Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discussion draft for a potential external pitch (JVM vendors, serviceability-dev): decompose FollowReferences/IterateOverReachableObjects into independently-callable root enumeration and per-object expansion primitives, so a caller can bound pause time and drive its own resumable traversal instead of being bound to one fused, all-or-nothing call. Deliberately kept separate from the internal VMStructs-based G1-only workaround this would let us retire if adopted. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…bugs in the manual reference-chains heap walk Correct oopMapBlockAt() to account for vtable/itable length before the OopMapBlock tail (root-caused via HotSpot source's start_of_itable()/ start_of_nonstatic_oop_maps()), fix arrayHeaderSizeBytes() double-applying a words-to-bytes conversion on a field layout_helper already reports in bytes, remove a -1 sentinel collision in arrayLayoutAvailable(), and replace the produced>0-OR-hasMore() resume condition (which spins forever on self-referential objects) with hasMore() alone in both the production ReferenceChainTracker callback and the FieldWalker test seam. Add sanity-bound asserts and JUnit test seams (FieldWalkerTestSeamsTest, IsG1ActiveTestSeamsTest) so a future JDK/vendor layout change fails loudly instead of silently misreading heap words as oops. Verified passing on Temurin/Microsoft JDK 21.x. JDK 25 Temurin has a separate, still-open uncompressed-oops-detection issue not addressed here. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d fields)
JDK-8329306 flattened CompressedOops's nested NarrowOopStruct, so the
vmStructs-exported field names dropped the "_narrow_oop." prefix
("_narrow_oop._base"/"_narrow_oop._shift" -> "_base"/"_shift") starting
in JDK 24. This repo's VMCompressedOops declaration only matched the old
prefixed names, so on JDK 24/25/26 the narrow-oop base/shift addresses
failed to resolve, isCompressedOopsEnabled() stayed false, and
FieldWalker fell through to the uncompressed-oop 8-byte read path,
misdecoding oop fields (e.g. reading garbage instead of null). Match
both spellings, the same fallback pattern already used for
CompressedKlassPointers's analogous fields.
Verified: FieldWalkerTestSeamsTest + IsG1ActiveTestSeamsTest pass on
JDK 21 (no regression), 25.0.3 Temurin, 26.0.1 Temurin, and 26.0.1
Liberica.
Environment: Datadog workspace
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…exported under Universe, not CompressedOops)
Before JDK 17 split CompressedOops out into its own AllStatic class,
HotSpot exported the narrow-oop base/shift as Universe statics
("_narrow_oop._base"/"_narrow_oop._shift"). VMCompressedOops's
MATCH_SYMBOLS only matched type "CompressedOops", so on JDK 8-16 these
fields never resolved and verify_offsets()'s hard VERIFY_FIELD assert
SIGABRT'd any debug-native test run.
VMUniverse already claims all "Universe"-typed entries (resolveOffsets()'s
scan continues after the first type_begin() match for a given exported
type name), so a separate later block matching "Universe" would never see
them. Add the narrow-oop fields directly to VMUniverse under distinct
names, version-gated to JDK 8-16, and fall back to them in
resolveOffsets() when the CompressedOops-sourced addresses are null.
Verified: full debug test suite (JDK 8) and FieldWalkerTestSeamsTest +
IsG1ActiveTestSeamsTest (JDK 11, 17, 21, 25.0.3 Temurin, 26.0.1 Temurin,
26.0.1 Liberica) all pass, confirming JDK 8 was not independently broken
by this assert and no regression on JDK 17+.
Environment: Datadog workspace
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The manual walk was previously gated on isG1Active() as a conservative
simplification, but it reads no G1-specific field - it only needs a
stable oop/klass snapshot during the JVMTI safepoint window, which
holds under any STW-only collector (Serial/Parallel/G1/Shenandoah), not
just G1. Only ZGC's concurrent relocation across that safepoint window
is genuinely unsafe for it. Add VMStructs::isZgcActive() (mirrors
isG1Active()'s VMFlag::find("UseZGC") pattern) and switch the
runPass() dispatch gate from isG1Active() && FieldWalker::available()
to !isZgcActive() && FieldWalker::available().
Verified via compare-refchains-repro.sh under Serial/G1/Shenandoah
(consistent ~10% overhead, sub-100ms pauses) and real ZGC on JDK 25
(correctly falls through to the JVMTI-only path instead, confirmed by
its much larger pause signature).
Environment: Datadog workspace
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The JVMTI walk is a poor fallback under low-latency collectors: on ZGC it imposes STW pauses roughly 3 orders of magnitude longer than the manual VMStructs walk (max round latency ~1.3s vs sub-100ms in comparison runs), which defeats the point of using a low-latency collector in the first place. Silently falling back to it whenever the manual walk is unavailable (ZGC active, or FieldWalker failed to resolve an offset) is the wrong default. Add referencechains=...:allowjvmtifallback=BOOL (default false, parsed in arguments.cpp like the other boolean sub-options). Without it, when the manual walk isn't available, reference chain tracking prints one loud, unconditional fprintf(stderr, ...) warning at start() and then never runs the JVMTI walk - passes are skipped with truncated=true (not false) so has_pending_frontier stays true and the search state remains RUNNING rather than flipping to COMPLETED, which would otherwise make shouldRunPass() retry restartSearch() every cadence tick forever for no benefit, since GC choice and FieldWalker's resolved offsets are both fixed for the life of the JVM and this gate can never change mid-run. The warning uses fprintf(stderr, ...) directly rather than Log::warn(): Log's default level is LOG_NONE (log.cpp), so Log::warn() is silently dropped unless the operator has separately set log=/loglevel=, which would defeat the 'very loud' requirement. This matches the existing convention for unconditional operator-facing warnings (see the JVMTI wallclock-unavailable warning in profiler.cpp). Verified on real ZGC (JDK 25.0.3-tem): without the flag, the warning fires once and zero chains are produced; with allowjvmtifallback=true, chains are produced and the round-latency signature matches the previously measured ZGC/JVMTI-fallback overhead. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Needed these to A/B the manual-walk gate change across collectors without relying on JVM ergonomic defaults: REFCHAINS_JAVA_HOME picks the JDK to launch with (e.g. to reproduce a version-specific VMStructs issue), and REFCHAINS_GC forces a specific collector via -XX:+Use<name>GC. compare-refchains-repro.sh passes both through unchanged to its on/off variants and prints them in the report header so results stay attributable to a specific JDK+GC combo. Used this to generate the Serial/G1/Shenandoah/ZGC comparison data cited in the adjacent VMStructs gate-change and JVMTI-fallback opt-in commits. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
referenceChains.cpp used VMKlass::cast (via VMKlass::fromOop), which instantiates cast_to<VMKlass>. That template is declared in vmStructs.h but only defined in vmStructs.inline.h, which this file was not including. No other translation unit instantiated the VMKlass specialization, so the linker failed with an undefined reference during the CodeQL Java/Kotlin autobuild. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341): a JVMTI
FollowReferences-basedBFS walk (
ReferenceChainTracker) tags reachable objects and recordsreferrer-type chains back toward a GC root, bridged to
LivenessTracker::selectLeakCandidates()'s population-slope leak detection.This revision also closes a structural gap found while adding end-to-end
coverage: a one-shot search could finish walking the whole reachable graph
before population-trend detection had accumulated enough GC epochs to flag a
candidate, making any object allocated after that point permanently
undiscoverable. Search restarts are now gated on an actual leak indication
existing (so a restarted walk is very likely to still find the object) plus
a
PainBudgetcooldown - a leaky bucket over cumulative safepoint timespent, not a fixed interval, so a cheap search can restart sooner than an
expensive one.
Motivation:
PROF-15341 - give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive".
Additional Notes:
See
doc/architecture/LiveHeapReferenceChains.mdanddoc/architecture/LiveHeapReferenceChains-RemainingWorkPlan.mdfor the fulldesign history, open questions, and what's still provisional (per-pass
budget/hop-cap/TTL/frontier-cap/pain-budget-refill-rate defaults are all
round, unbenchmarked placeholders pending a future measurement pass).
How to test the change?:
./gradlew :ddprof-lib:gtestDebug- full native unit suite, including newPainBudgetTest/SearchRestartTestcoverage for the restart gate../gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*"-genuinely separate-process end-to-end test: runs a real leaking-cache Java
app in a child JVM and asserts the reconstructed chain's leaf class.
./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*"-in-process coverage for the BFS engine, target-selection bridging, and
abandonment reporting.
For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!
🤖 Generated with Claude Code