Reference chains for surviving live-heap samples via a bounded manual heap walk (PROF-15341) - #644
Draft
jbachorik wants to merge 7 commits into
Draft
Reference chains for surviving live-heap samples via a bounded manual heap walk (PROF-15341)#644jbachorik wants to merge 7 commits into
jbachorik wants to merge 7 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
Contributor
CI Test ResultsRun: #31107126177 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-08-06 14:04:07 UTC |
Contributor
Reliability & Chaos Results❌ 1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/127431974 ❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos |
Contributor
Benchmark Results (commit cfec595)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1949 / 2055 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8727 / 8389 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8166 / 8396 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1292 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1288 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2830 / 3023 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2976 / 2821 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | · / ✅ | · / ✅ | · / 3 | · / 3555 | · / ✅ | · / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3499 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1592 / 1697 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1922 / 1854 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 10, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 0714bd3)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10184 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +1.8% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8891 ms (24 iters) | ✅ 8865 ms (24 iters) | ≈ -0.3% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6003 ms (33 iters) | ✅ 6034 ms (33 iters) | ≈ +0.5% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5507 ms (36 iters) | ✅ 5426 ms (36 iters) | ≈ -1.5% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2834 ms (66 iters) | ✅ 2761 ms (67 iters) | 🟢 -2.6% | — / — |
| fj-kmeans | 25 | ✅ 2824 ms (66 iters) | ✅ 2838 ms (66 iters) | ≈ +0.5% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2113 ms (88 iters) | ✅ 2065 ms (89 iters) | ≈ -2.3% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2059 ms (90 iters) | ✅ 2102 ms (89 iters) | ≈ +2.1% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1283 ms (134 iters) | ✅ 1260 ms (135 iters) | ≈ -1.8% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 16658 ms (15 iters) | ✅ 16371 ms (16 iters) | ≈ -1.7% (±9%) | — / — |
| reactors | 25 | ✅ 18507 ms (15 iters) | ✅ 18056 ms (15 iters) | ≈ -2.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1929 / 2034 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2310 / 2113 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 8529 / 8852 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8635 / 8390 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 1284 / 1281 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 1283 / 1285 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2930 / 2864 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2963 / 2953 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3558 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3463 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1564 / 1782 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1681 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 700d838)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275960 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10425 ms (21 iters) | ✅ 10259 ms (21 iters) | ≈ -1.6% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8748 ms (24 iters) | ✅ 8797 ms (24 iters) | ≈ +0.6% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5970 ms (33 iters) | ✅ 5982 ms (33 iters) | ≈ +0.2% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5468 ms (36 iters) | ≈ -0% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2598 ms (73 iters) | ✅ 2715 ms (69 iters) | 🔴 +4.5% | — / — |
| fj-kmeans | 25 | ✅ 2784 ms (67 iters) | ✅ 2847 ms (66 iters) | ≈ +2.3% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2122 ms (87 iters) | ✅ 2126 ms (88 iters) | ≈ +0.2% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2035 ms (91 iters) | ✅ 2057 ms (91 iters) | ≈ +1.1% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1304 ms (131 iters) | ✅ 1253 ms (136 iters) | ≈ -3.9% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 1018 ms (168 iters) | ≈ +0.3% (±31.8%) | — / — |
| reactors | 21 | ✅ 16107 ms (15 iters) | ✅ 16847 ms (15 iters) | ≈ +4.6% (±7.9%) | — / — |
| reactors | 25 | ✅ 18469 ms (15 iters) | ✅ 19075 ms (14 iters) | ≈ +3.3% (±4.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2228 / 2453 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8686 / 8417 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8561 / 8290 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1272 / 1289 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1287 / 1298 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2912 / 3017 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2907 / 2872 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3515 / 3489 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 7 | 3498 / 3463 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1732 / 1701 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1780 / 1924 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 14, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 14, 2026 12:03
700d838 to
9972bff
Compare
Contributor
Benchmark Results (commit 9972bff)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124511782 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a636398)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124700626 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit f2d8978)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124747648 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a48899b)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124773626 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10376 ms (21 iters) | ✅ 10228 ms (21 iters) | ≈ -1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8829 ms (24 iters) | ✅ 8859 ms (24 iters) | ≈ +0.3% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 6001 ms (33 iters) | ✅ 5928 ms (33 iters) | ≈ -1.2% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5475 ms (36 iters) | ≈ +0.1% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2687 ms (70 iters) | ✅ 2761 ms (68 iters) | 🔴 +2.8% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2805 ms (66 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2047 ms (91 iters) | ✅ 2116 ms (88 iters) | 🔴 +3.4% | — / — |
| future-genetic | 25 | ✅ 2000 ms (93 iters) | ✅ 2083 ms (89 iters) | 🔴 +4.2% | — / — |
| naive-bayes | 21 | ✅ 1256 ms (136 iters) | ✅ 1253 ms (136 iters) | ≈ -0.2% (±32.8%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1016 ms (169 iters) | ≈ 0% (±31.3%) | — / — |
| reactors | 21 | ✅ 16870 ms (15 iters) | ✅ 16463 ms (15 iters) | ≈ -2.4% (±7.3%) | — / — |
| reactors | 25 | ✅ 18270 ms (15 iters) | ✅ 18333 ms (15 iters) | ≈ +0.3% (±4.5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1988 / 2024 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2263 / 2395 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 6 | 8440 / 8679 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8283 / 8410 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1288 / 1260 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 1250 / 1295 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2968 / 3003 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2843 / 2902 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3474 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3495 / 3488 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1726 / 1721 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1893 / 1750 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 787f29e)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124811024 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10215 ms (21 iters) | ✅ 10475 ms (21 iters) | ≈ +2.5% (±11.9%) | — / — |
| akka-uct | 25 | ✅ 8843 ms (24 iters) | ✅ 8861 ms (24 iters) | ≈ +0.2% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ -0.3% (±25.5%) | |
| finagle-chirper | 25 | ✅ 5515 ms (36 iters) | ✅ 5479 ms (36 iters) | ≈ -0.7% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2693 ms (70 iters) | ✅ 2836 ms (66 iters) | 🔴 +5.3% | — / — |
| fj-kmeans | 25 | ✅ 2820 ms (66 iters) | ✅ 2837 ms (66 iters) | ≈ +0.6% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2059 ms (90 iters) | ✅ 2070 ms (89 iters) | ≈ +0.5% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2089 ms (89 iters) | ✅ 2130 ms (87 iters) | ≈ +2% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1284 ms (133 iters) | ✅ 1313 ms (131 iters) | ≈ +2.3% (±33%) | — / — |
| naive-bayes | 25 | ✅ 1009 ms (170 iters) | ✅ 988 ms (173 iters) | ≈ -2.1% (±31.4%) | — / — |
| reactors | 21 | ✅ 16540 ms (15 iters) | ✅ 16155 ms (15 iters) | ≈ -2.3% (±7.1%) | — / — |
| reactors | 25 | ✅ 18545 ms (15 iters) | ✅ 18816 ms (15 iters) | ≈ +1.5% (±4.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1987 / 1990 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2238 / 2323 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8290 / 8529 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8323 / 8386 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1299 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1274 / 1299 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2920 / 2982 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2886 / 2864 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3505 / 3499 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1593 / 1669 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1849 / 1940 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit fede524)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124833099 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit ef0f5c6)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124961539 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 1299316)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124967829 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10336 ms (21 iters) | ✅ 10245 ms (21 iters) | ≈ -0.9% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8868 ms (24 iters) | ✅ 8789 ms (24 iters) | ≈ -0.9% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 5968 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ +0.1% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5510 ms (36 iters) | ✅ 5537 ms (36 iters) | ≈ +0.5% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2696 ms (69 iters) | ✅ 2822 ms (66 iters) | 🔴 +4.7% | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2746 ms (68 iters) | 🟢 -3.4% | — / — |
| future-genetic | 21 | ✅ 2068 ms (90 iters) | ✅ 2051 ms (90 iters) | ≈ -0.8% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2134 ms (87 iters) | ✅ 2059 ms (90 iters) | 🟢 -3.5% | — / — |
| naive-bayes | 21 | ✅ 1318 ms (130 iters) | ✅ 1253 ms (136 iters) | ≈ -4.9% (±31.8%) | — / — |
| naive-bayes | 25 | ✅ 998 ms (171 iters) | ✅ 1025 ms (167 iters) | ≈ +2.7% (±32.1%) | — / — |
| reactors | 21 | ✅ 16264 ms (15 iters) | ✅ 16625 ms (15 iters) | ≈ +2.2% (±7.4%) | — / — |
| reactors | 25 | ✅ 17926 ms (15 iters) | ✅ 18917 ms (15 iters) | ≈ +5.5% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1917 / 1934 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 2285 / 2434 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8503 / 8628 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8331 / 8664 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1287 / 1266 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1277 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 3037 / 2981 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2897 / 2943 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 12 | 3507 / 3493 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 5 | 3446 / 3481 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1765 / 1665 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1855 / 1920 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 16, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 16, 2026 11:31
1299316 to
baba070
Compare
Contributor
Benchmark Results (commit baba070)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124983919 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 35c49c2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997895 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10367 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +0% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8917 ms (24 iters) | ✅ 8766 ms (24 iters) | ≈ -1.7% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5972 ms (33 iters) | ≈ -0.2% (±25.3%) | |
| fj-kmeans | 21 | ✅ 2801 ms (67 iters) | ✅ 2750 ms (68 iters) | ≈ -1.8% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2756 ms (68 iters) | 🟢 -3% | — / — |
| future-genetic | 21 | ✅ 2142 ms (87 iters) | ✅ 2080 ms (89 iters) | 🟢 -2.9% | — / — |
| future-genetic | 25 | ✅ 1991 ms (94 iters) | ✅ 2035 ms (91 iters) | ≈ +2.2% (±2.7%) | — / — |
| naive-bayes | 25 | ✅ 1025 ms (167 iters) | ✅ 1010 ms (169 iters) | ≈ -1.5% (±31.4%) | — / — |
| reactors | 21 | ✅ 16244 ms (15 iters) | ✅ 16474 ms (15 iters) | ≈ +1.4% (±7.9%) | — / — |
| reactors | 25 | ✅ 18288 ms (15 iters) | ✅ 18332 ms (15 iters) | ≈ +0.2% (±3.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2303 / 2180 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 8767 / 8881 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8034 / 8300 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1316 / 1287 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1289 / 1282 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2889 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2935 / 2838 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3496 / 3495 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3477 / 3510 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1642 / 1586 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1802 / 1759 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit a9fb05f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125008807 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10453 ms (21 iters) | ≈ +1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8814 ms (24 iters) | ✅ 8779 ms (24 iters) | ≈ -0.4% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 6006 ms (33 iters) | ✅ 6047 ms (33 iters) | ≈ +0.7% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5467 ms (36 iters) | ✅ 5455 ms (36 iters) | ≈ -0.2% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2772 ms (68 iters) | ✅ 2687 ms (70 iters) | 🟢 -3.1% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2801 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2087 ms (88 iters) | ✅ 2055 ms (90 iters) | ≈ -1.5% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2088 ms (89 iters) | ✅ 2046 ms (91 iters) | ≈ -2% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1233 ms (139 iters) | ✅ 1257 ms (136 iters) | ≈ +1.9% (±32.9%) | — / — |
| naive-bayes | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
| reactors | 21 | ✅ 16947 ms (15 iters) | ✅ 15936 ms (15 iters) | ≈ -6% (±8.2%) | — / — |
| reactors | 25 | ✅ 18681 ms (15 iters) | ✅ 18731 ms (15 iters) | ≈ +0.3% (±5.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2008 / 2140 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2058 / 2262 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8437 / 8783 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8064 / 8228 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 6 / ✅ | 1275 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1250 / 1261 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2876 / 3036 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2912 / 2869 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3492 / 3491 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1892 / 1633 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1914 / 1937 | ✅ / ✅ | ✅ / ✅ |
3 tasks
Contributor
Benchmark Results (commit 2033006)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125072159 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10363 ms (21 iters) | ✅ 10147 ms (21 iters) | ≈ -2.1% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8966 ms (24 iters) | ✅ 8873 ms (24 iters) | ≈ -1% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6007 ms (33 iters) | ✅ 5986 ms (33 iters) | ≈ -0.3% (±26.1%) | |
| finagle-chirper | 25 | ✅ 5488 ms (36 iters) | ✅ 5433 ms (36 iters) | ≈ -1% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2726 ms (68 iters) | ✅ 2760 ms (68 iters) | ≈ +1.2% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2848 ms (66 iters) | ✅ 2743 ms (69 iters) | 🟢 -3.7% | — / — |
| future-genetic | 21 | ✅ 2062 ms (90 iters) | ✅ 2047 ms (90 iters) | ≈ -0.7% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1994 ms (93 iters) | ✅ 2057 ms (90 iters) | 🔴 +3.2% | — / — |
| naive-bayes | 21 | ✅ 1269 ms (135 iters) | ✅ 1258 ms (136 iters) | ≈ -0.9% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1016 ms (168 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16554 ms (15 iters) | ✅ 16630 ms (15 iters) | ≈ +0.5% (±8.5%) | — / — |
| reactors | 25 | ✅ 18407 ms (15 iters) | ✅ 18308 ms (15 iters) | ≈ -0.5% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2101 / 2026 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 2336 / 2365 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8772 / 8497 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8397 / 8470 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1260 / 1274 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1303 / 1307 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2893 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2958 / 2912 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3540 / 3527 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3458 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1736 / 1754 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1990 / 1872 | ✅ / ✅ | ✅ / ✅ |
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 17, 2026 11:39
2033006 to
02a0a93
Compare
Contributor
Benchmark Results (commit 02a0a93)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125255579 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Bits has a CI fix ready🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready
View in Datadog | Reviewed commit ed6525a · Any feedback? Reach out in #deveng-pr-agent |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 59 changed files in this pull request and generated no new comments.
Suppressed comments (12)
utils/sweep-refchains-budgets.sh:93
- Because the script runs with
set -euo pipefail, thisstw_stats=$(grep ... | ... | awk ...)assignment will abort the whole sweep when the first grep finds no matches (exit 1) — making the intended fallback format branch unreachable.
utils/sweep-refchains-budgets.sh:97 - Same issue as the JDK17+-format branch: without making the pipeline non-fatal, a missing match will terminate the script (and potentially skip emitting a row) instead of producing the documented 0/empty defaults.
utils/sweep-refchains-all.sh:82 - This
stw_stats=$(grep ... | ... | awk ...)pipeline can cause the sweep to exit early underset -euo pipefailwhen there are no matches for the JDK17+-style safepoint format (grep exit 1). That makes the fallback branch unreachable.
utils/sweep-refchains-all.sh:87 - Same pipefail/errexit issue as the JDK17+-format branch: if this grep finds no matches, the command substitution can terminate the script instead of producing default 0 STW metrics.
utils/compare-refchains-repro.sh:103 - Same issue as the JDK17+-format branch: without making this pipeline non-fatal, a missing match can abort the whole comparison run instead of falling back to
0 0 0 0.
utils/sweep-refchains-budgets.sh:81 - With
set -euo pipefail, these metric-extraction pipelines will cause the script to exit if the[metrics]line is missing or a field regex doesn’t match (grep returns non-zero + pipefail). The intent appears to be “default to 0 and keep going” (as in compare-refchains-repro.sh), so the pipelines should be made non-fatal.
This issue also appears in the following locations of the same file:
- line 90
- line 95
utils/sweep-refchains-all.sh:76
- With
set -euo pipefail, these grep-based extractions will terminate the whole OFAT sweep if the[metrics]line or any expected key is missing. The rest of the function already defaults missing values to 0, so these pipelines should be made non-fatal to match that contract.
This issue also appears in the following locations of the same file:
- line 79
- line 84
utils/compare-refchains-repro.sh:99
- With
set -euo pipefail, thisstw_stats=$(grep ... | ... | awk ...)assignment can terminate the script when the JDK17+-style safepoint format isn’t present (grep exit 1 under pipefail). That makes the intended fallback format branch unreachable.
This issue also appears on line 101 of the same file.
utils/refchains-report.py:131
- If the CSV has only a header (or is empty),
rowswill be empty andrender_chart()will crash onmin(xs)/max(xs). It’s better to fail gracefully and still write an HTML file explaining there was no data.
utils/refchains-report-multi.py:66 - If the input CSV is empty (or contains only headers),
rowsis empty and later code will crash (e.g.,rows[0],min(xs_num)). Writing a minimal HTML report and returning makes failures much easier to diagnose.
doc/reference-chains-design.md:5 - The PR and description are tied to PROF-15341, but the design doc header still says
**Jira:** TBD, which makes it harder to trace provenance later.
ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTestSeamsTest.java:145 assertTrue(!target.equals(null))is a confusing way to keep the object reachable (and reads like it’s testingequals()semantics). A direct null check expresses the intent without invokingequals().
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 59 changed files in this pull request and generated no new comments.
Suppressed comments (9)
utils/sweep-refchains-budgets.sh:93
- The
stw_stats=$(grep ... | ... | awk ...)pipelines can also abort the sweep underset -euo pipefailwhen grep finds no matches (exit=1). That prevents the intended fallback to the JDK<=16 log format and makes the script sensitive to minor unified-logging format differences.
utils/sweep-refchains-all.sh:82 - The
stw_stats=$(grep ... | ... | awk ...)pipelines can abort the script underset -euo pipefailif grep finds no matches (exit=1), preventing the fallback parsing path and making the sweep brittle across JDK unified-logging variations.
utils/sweep-refchains-budgets.sh:81 - With
set -euo pipefail, these metric-extraction pipelines will terminate the whole sweep if the[metrics]line is missing/malformed (grep exits 1 -> pipeline failure). That makes the sweep brittle in the presence of a failed/aborted JVM run (where you’d prefer the row to default to 0s and continue).
This issue also appears on line 89 of the same file.
utils/sweep-refchains-all.sh:76
- With
set -euo pipefail, these pipelines will terminate the entire OFAT sweep if the[metrics]line is missing/malformed (grep exits 1 -> pipeline failure). It’s better to tolerate missing metrics (default to 0s) so failed points don’t abort the whole run matrix.
This issue also appears on line 78 of the same file.
utils/compare-refchains-repro.sh:125
- The report table header says the delta column is “on vs. off”, but throughput delta is computed as
pct_delta(off, on), which flips the sign/denominator relative to every other metric. This makes throughput’s delta inconsistent and easy to misinterpret.
ddprof-lib/src/main/cpp/arguments.h:39 - This comment references design/plan docs that are not present in the repository (
doc/architecture/LiveHeapReferenceChains-ImplementationPlan.md,-BenchmarkPlan.md,-RemainingWorkPlan.md). That makes the in-code rationale hard to follow and creates dead links.
// Every constant below is a provisional default pending empirical
// tuning (see doc/architecture/LiveHeapReferenceChains-ImplementationPlan.md)
// - none of these values are backed by a benchmark run against this
// codebase. Each is chosen conservatively from cited precedent or from the
// shape of an existing, already-tuned subsystem, per the rationale below;
utils/refchains-report.py:46
render_chart()callsmin(xs)/max(xs)unconditionally, which throws on an empty CSV (e.g., if the sweep exits early or is invoked with an empty value list). The caller already toleratesrows == []for the table section, so charts should also handle empty input gracefully.
ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTestSeamsTest.java:145assertTrue(!target.equals(null))is an unusual way to keep the reference alive; it also reads like a (misguided) null-check. Using a direct null check is clearer and still creates a use oftargetto keep it reachable to this point.
ddprof-lib/src/main/cpp/callTraceStorage.cpp:7#include <unistd.h>appears to be unused in this file (nousleep,getpid, etc.). Keeping includes minimal helps avoid portability surprises and reduces compile churn.
#include <unistd.h>
Contributor
Benchmark Results (commit 7fddcbc)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128879382 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | ✅ 8934 ms (24 iters) | ✅ 8841 ms (24 iters) | ≈ -1% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 5987 ms (33 iters) | ✅ 5938 ms (33 iters) | ≈ -0.8% (±25.6%) | |
| finagle-chirper | 25 | ✅ 5449 ms (36 iters) | ✅ 5536 ms (36 iters) | ≈ +1.6% (±24.6%) | |
| fj-kmeans | 21 | ✅ 2758 ms (68 iters) | ✅ 2742 ms (68 iters) | ≈ -0.6% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2860 ms (66 iters) | ✅ 2779 ms (68 iters) | 🟢 -2.8% | — / — |
| future-genetic | 21 | ✅ 2157 ms (87 iters) | ✅ 2065 ms (90 iters) | 🟢 -4.3% | — / — |
| future-genetic | 25 | ✅ 2040 ms (91 iters) | ✅ 2128 ms (87 iters) | 🔴 +4.3% | — / — |
| naive-bayes | 21 | ✅ 1271 ms (135 iters) | ✅ 1276 ms (135 iters) | ≈ +0.4% (±32.4%) | — / — |
| naive-bayes | 25 | ✅ 1020 ms (168 iters) | ✅ 1014 ms (169 iters) | ≈ -0.6% (±31.5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 4 | 2302 / 2364 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 7 | 8704 / 8781 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8443 / 8475 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 1 | 1265 / 1284 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 5 | 1285 / 1298 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 4 | 3016 / 2863 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2845 / 2988 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 3539 / 3522 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 3523 / 3496 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 1578 / 1605 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1814 / 1799 | ✅ / ✅ | ✅ / ✅ |
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 6, 2026 10:07
7fddcbc to
351fb75
Compare
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 6, 2026 10:23
351fb75 to
34b95e1
Compare
Contributor
Benchmark Results (commit eecff95)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/129368391 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10313 ms (21 iters) | ✅ 10355 ms (21 iters) | ≈ +0.4% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5990 ms (33 iters) | ✅ 5967 ms (33 iters) | ≈ -0.4% (±24.7%) | |
| finagle-chirper | 25 | ✅ 5408 ms (36 iters) | ✅ 5394 ms (36 iters) | ≈ -0.3% (±23.2%) | |
| fj-kmeans | 21 | ✅ 2810 ms (66 iters) | ✅ 2816 ms (66 iters) | ≈ +0.2% (±2.4%) | — / — |
| fj-kmeans | 25 | ✅ 2855 ms (66 iters) | ✅ 2838 ms (66 iters) | ≈ -0.6% (±2.5%) | — / — |
| future-genetic | 21 | ✅ 2052 ms (91 iters) | ✅ 2130 ms (87 iters) | 🔴 +3.8% | — / — |
| future-genetic | 25 | ✅ 1890 ms (98 iters) | ✅ 1936 ms (96 iters) | ≈ +2.4% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1306 ms (131 iters) | ✅ 1242 ms (137 iters) | ≈ -4.9% (±31.5%) | — / — |
| naive-bayes | 25 | ✅ 1014 ms (168 iters) | ✅ 1013 ms (169 iters) | ≈ -0.1% (±31.3%) | — / — |
| reactors | 21 | ✅ 15853 ms (15 iters) | ✅ 16099 ms (15 iters) | ≈ +1.6% (±8.4%) | — / — |
| reactors | 25 | ✅ 18622 ms (15 iters) | ✅ 18557 ms (15 iters) | ≈ -0.3% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2030 / 2005 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2158 / 2298 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 5 | 8646 / 8625 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8200 / 8102 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1283 / 1269 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1298 / 1271 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2928 / 2865 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2866 / 2954 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 4 | 3499 / 3473 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 3471 / 3517 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1564 / 1651 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1830 / 1897 | ✅ / ✅ | ✅ / ✅ |
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 6, 2026 12:36
eecff95 to
92d65f2
Compare
…chains Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a manual VMStructs-based heap walk that discovers reference chains from GC roots to live-heap sample objects, recorded as datadog.ReferenceChain / datadog.ReferenceChainAbandoned JFR events. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reset class-count counters in restartSearch(), fix null-base pointer UB in symbols_linux.cpp, gate JNI ref-minting in foldKlassCountsLocked() on allow_resolve, add totalHops to the ReferenceChain JFR event to avoid silent truncation, make LivenessTracker::_gc_generations atomic, widen per-klass population counters to u32, restore deleted FrameType tests, and correct stale root-discovery comments/docs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes the CodeQL Autobuild failure: this test still used JMC's IItemCollection/IItem after AbstractProfilerTest's verifyEvents() switched to the jafar-backed JfrEvents API. Adds a JfrEvents overload of ReferenceChainAssertions.findMatchForClass alongside the existing IItemCollection one, which LeakingCacheScenario/ReferenceChainJfrParserTest still use unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 6, 2026 13:41
92d65f2 to
da35063
Compare
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341). A
ReferenceChainTrackerruns adedicated BFS thread that tags reachable objects with JVMTI object tags,
walks the heap incrementally across GC epochs, and records the referrer-type
chain back toward a GC root (labelled with its GC-root kind). It is bridged
to
LivenessTracker::selectLeakCandidates()'s population-slope leakdetection: when a still-live sampled object starts to look leak-shaped, its
chain is reconstructed from the frontier table and emitted to JFR.
flowchart TD LT["LivenessTracker::selectLeakCandidates (population-slope ranking)"] -->|"ranked klass candidates"| PWT["ReferenceChainTracker::pollWatchedTargets"] BFS["BFS thread: threadLoop"] -->|"shouldRunPass gate: GC epoch advanced or cadence elapsed"| RP["runPass"] RP -->|"every pass"| MW["runPassManualWalk: IterateOverReachableObjects seeds roots"] MW --> EF["expandFrontier: batched array-holder FollowReferences"] EF --> FT["FrontierTable: FRONTIER / EXPANDED / EDGE / ABANDONED"] FT -->|"tag already set on a leak-candidate instance"| PWT PWT -->|"buildChainEvent"| RC["cacheResolvedChain: one entry per klass id, cap 256"] RC -->|"Profiler::dump, snapshot without clearing"| DR["drainPendingChainEvents"] DR --> JFR["datadog.ReferenceChain / datadog.ReferenceChainAbandoned"]Heap-walk mechanism (the main design decision in this branch): every pass
takes a bounded manual walk (
runPassManualWalk()) — pure JVMTI(
IterateOverReachableObjectsfor root enumeration +expandFrontier()forincremental frontier expansion), resumable across passes, and bounded so its
safepoint pauses stay small on every collector, including ZGC. The walk
issues only JVMTI heap calls, which run inside the
VM_HeapWalkOperationsafepoint and honor ZGC's load barriers, so concurrent relocation cannot
corrupt it — it reads no raw oop.
expandFrontier()'s batching, concretely: instead of oneFollowReferencescall per frontier entry, up to
budgetpending tags are resolved in oneGetObjectsWithTagscall, packed into a single JNI array (holder), andexpanded via exactly one
FollowReferences(initial_object = holder)call —so one BFS level is discovered per VM-safepoint operation rather than one
safepoint per entry:
sequenceDiagram participant BFS as BFS thread participant JVMTI as JVMTI participant JNI as JNI holder array participant CB as heapReferenceCallback BFS->>BFS: pull up to budget tags from front of _pending_expand BFS->>JVMTI: GetObjectsWithTags, resolve which tags are still live JVMTI-->>BFS: live jobject references BFS->>JNI: EnsureLocalCapacity, then NewObjectArray to build holder alt exception, EnsureLocalCapacity failure, or null holder BFS->>BFS: ctx.truncated = true, retry this batch next pass else holder built successfully BFS->>JNI: SetObjectArrayElement per resolved object BFS->>JVMTI: FollowReferences, initial_object = holder JVMTI->>CB: heapReferenceCallback per outgoing edge CB-->>JVMTI: descend only for batch_tags boundary objects JVMTI-->>BFS: one BFS hop expanded for the whole batch BFS->>BFS: markExpanded, admitObject appends children to _pending_expand endexpandFrontier()'s JNI/JVMTI error handling is defensive by construction: anull
holder, a pending JNI exception afterNewObjectArray/SetObjectArrayElement, or anEnsureLocalCapacityfailure all setctx.truncated = true(retry next pass) instead of marking the batchpermanently
EXPANDED; a failedjava/lang/Objectclass resolution withpending work also forces
truncated = true, sorunPass()cannot mistakeit for
SearchState::COMPLETED.Termination and pacing — an unbounded traversal could otherwise stall a
GC safepoint or run forever:
stateDiagram-v2 direction LR [*] --> RUNNING RUNNING --> COMPLETED: frontier drained,<br/>no truncation this pass RUNNING --> ABANDONED: frontier-size cap hit,<br/>or wall-clock TTL exceeded<br/>with work still pending COMPLETED --> RUNNING: restartSearch ABANDONED --> RUNNING: restartSearch note right of ABANDONED SearchAbandonReason (FRONTIER_CAP/TTL) records which cutoff fired, for the T_REFERENCE_CHAIN_ABANDONED JFR event end note note right of RUNNING restartSearch only fires once a leak candidate is seen and PainBudget allows it end notetreated as immediate search abandonment, not per-pass truncation) —
configured via
referencechains=true:hops=N:budget=N:ttl=N:framecap=N(plus
firstpassbudget=Nfor the first pass's own budget). Negativehops/budget/framecapvalues are floored (an unfloored negativehopswould otherwise wrap to ~4e9 as au32, silently disabling the capit's meant to enforce); all three are also ceiling-clamped.
updatePacing(),pausetarget=Nms)adapts the effective per-pass budget and cadence toward a configured
safepoint-pause SLO.
restarted walk is very likely to still find the object) plus a
PainBudgetcooldown (painbudget=N, 0-100) — a leaky bucket overcumulative safepoint time spent, so a cheap search can restart sooner than
an expensive one. This closes a structural gap where a one-shot walk could
finish before population-trend detection had accumulated enough GC epochs
to flag a candidate, leaving anything allocated afterward permanently
undiscoverable.
LivenessTracker::selectLeakCandidates()) requiresa klass's population growth to clear both a magnitude bar and a floor-rise
bar for
LEAK_TREND_HYSTERESIS_BASE(5) consecutive qualifying epochs —lowered to
LEAK_TREND_HYSTERESIS_CORROBORATED(3) when the aggregatepost-GC heap floor is also rising — rather than trusting a single epoch's
positive slope. This closes a false-positive gap where an oscillating
("see-saw") population could otherwise trigger a search with no real
longer-term growth.
JFR persistence across dumps: a resolved chain is cached per klass id
(
_resolved_chains, capped at 256 entries, drop-not-evict once full — seeREFERENCE_CHAIN_EVENTS_DROPPED) and re-stamped into every subsequent dumpthe sample survives into (
drainPendingChainEvents()snapshots withoutclearing, mirroring how
LivenessTrackerre-emits live-object samples), so along-lived leak's chain is present in each JFR chunk rather than only the
chunk active when it was first reconstructed. Verified end-to-end in
ExternalProcessReferenceChainTest.Motivation:
PROF-15341 — give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive",
without regressing safepoint-pause behaviour on low-pause collectors.
Additional Notes:
See
doc/reference-chains-design.mdanddoc/reference-chains-collection-summary.mdfor the full design and leak-detection mechanism. Per-pass
budget/hop-cap/TTL/frontier-cap/pause-target/pain-budget defaults are round,
unbenchmarked placeholders pending a measurement pass — no JMH harness for
this feature exists yet, though
utils/has repro, parameter-sweep, andJFR-report shell/Python tooling used to characterise pause behaviour ad hoc.
Earlier planning/proposal docs (implementation plan, remaining-work plan,
benchmark plan, and an alternative VMStructs-walk design) described approaches
or work that were superseded or shipped differently than planned, so they were
dropped from this PR (kept locally, not committed) rather than left to drift
further from the code.
How to test the change?:
./gradlew :ddprof-lib:gtestDebug— native unit suite, includingReferenceChainsTest/ReferenceChainsBfsTest/ReferenceChainsTagTest/FrontierTableTest/PollWatchedTargetsTest/ResolvedChainCacheTest,PainBudgetTest,SearchRestartTest(restart gate),ReferenceChainJfrRoundtripTest(JFR encode/decode),ArgumentsTest(
referencechains=sub-option parsing, includinghops/budget/framecapclamping behaviour), and the
LivenessTracker/SelectLeakCandidatesbridging tests.
./gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*"—genuinely separate-process end-to-end test: runs a real leaking-cache Java
app in a child JVM, asserts the reconstructed chain's leaf class, and
confirms the chain re-emits into three fresh JFR dumps (across-dumps
persistence).
./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*"—in-process coverage for the walk engine, target-selection bridging, and
abandonment reporting.
For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!
🤖 Generated with Claude Code