Nikita.tkachenko/runtime code coverage#10951
Draft
nikita-tkachenko-datadog wants to merge 19 commits intomasterfrom
Draft
Nikita.tkachenko/runtime code coverage#10951nikita-tkachenko-datadog wants to merge 19 commits intomasterfrom
nikita-tkachenko-datadog wants to merge 19 commits intomasterfrom
Conversation
Implement Coverage Binary Protocol v1 (CoverageBinaryEncoder) using two bit vectors per record for executable and covered lines. Switch from LCOV text format to the new "ddcov" binary format for coverage uploads. Add className to ClassProbeMapping, key coverage data by CoverageKey (sourceFile + className), and include language/env tags in upload metadata. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 63 metrics, 8 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~3b91f88621, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.065 s) : 0, 1065258
Total [baseline] (11.133 s) : 0, 11133116
Agent [candidate] (1.056 s) : 0, 1056083
Total [candidate] (10.978 s) : 0, 10977611
section appsec
Agent [baseline] (1.252 s) : 0, 1251888
Total [baseline] (11.257 s) : 0, 11256731
Agent [candidate] (1.246 s) : 0, 1246151
Total [candidate] (11.149 s) : 0, 11149227
section iast
Agent [baseline] (1.23 s) : 0, 1229842
Total [baseline] (11.424 s) : 0, 11424453
Agent [candidate] (1.237 s) : 0, 1236872
Total [candidate] (11.319 s) : 0, 11318673
section profiling
Agent [baseline] (1.19 s) : 0, 1189564
Total [baseline] (11.061 s) : 0, 11061270
Agent [candidate] (1.192 s) : 0, 1192423
Total [candidate] (11.066 s) : 0, 11065613
gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~3b91f88621, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.192 ms) : 0, 1192
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (634.514 ms) : 0, 634514
BytebuddyAgent [candidate] (618.153 ms) : 0, 618153
AgentMeter [baseline] (29.746 ms) : 0, 29746
AgentMeter [candidate] (29.327 ms) : 0, 29327
GlobalTracer [baseline] (258.414 ms) : 0, 258414
GlobalTracer [candidate] (256.894 ms) : 0, 256894
AppSec [baseline] (31.838 ms) : 0, 31838
AppSec [candidate] (31.627 ms) : 0, 31627
Debugger [baseline] (60.513 ms) : 0, 60513
Debugger [candidate] (60.363 ms) : 0, 60363
Remote Config [baseline] (588.628 µs) : 0, 589
Remote Config [candidate] (597.055 µs) : 0, 597
Telemetry [baseline] (8.015 ms) : 0, 8015
Telemetry [candidate] (8.213 ms) : 0, 8213
Flare Poller [baseline] (4.281 ms) : 0, 4281
Flare Poller [candidate] (3.561 ms) : 0, 3561
section appsec
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.203 ms) : 0, 1203
BytebuddyAgent [baseline] (661.793 ms) : 0, 661793
BytebuddyAgent [candidate] (649.275 ms) : 0, 649275
AgentMeter [baseline] (12.197 ms) : 0, 12197
AgentMeter [candidate] (12.065 ms) : 0, 12065
GlobalTracer [baseline] (258.752 ms) : 0, 258752
GlobalTracer [candidate] (257.171 ms) : 0, 257171
IAST [baseline] (24.214 ms) : 0, 24214
IAST [candidate] (24.044 ms) : 0, 24044
AppSec [baseline] (177.693 ms) : 0, 177693
AppSec [candidate] (177.276 ms) : 0, 177276
Debugger [baseline] (66.948 ms) : 0, 66948
Debugger [candidate] (65.894 ms) : 0, 65894
Remote Config [baseline] (641.461 µs) : 0, 641
Remote Config [candidate] (628.467 µs) : 0, 628
Telemetry [baseline] (8.472 ms) : 0, 8472
Telemetry [candidate] (8.369 ms) : 0, 8369
Flare Poller [baseline] (3.68 ms) : 0, 3680
Flare Poller [candidate] (3.626 ms) : 0, 3626
section iast
crashtracking [baseline] (1.185 ms) : 0, 1185
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (797.257 ms) : 0, 797257
BytebuddyAgent [candidate] (791.624 ms) : 0, 791624
AgentMeter [baseline] (11.412 ms) : 0, 11412
AgentMeter [candidate] (11.67 ms) : 0, 11670
GlobalTracer [baseline] (247.877 ms) : 0, 247877
GlobalTracer [candidate] (248.448 ms) : 0, 248448
IAST [baseline] (25.379 ms) : 0, 25379
IAST [candidate] (25.546 ms) : 0, 25546
AppSec [baseline] (28.378 ms) : 0, 28378
AppSec [candidate] (26.769 ms) : 0, 26769
Debugger [baseline] (69.16 ms) : 0, 69160
Debugger [candidate] (69.061 ms) : 0, 69061
Remote Config [baseline] (529.367 µs) : 0, 529
Remote Config [candidate] (532.527 µs) : 0, 533
Telemetry [baseline] (9.241 ms) : 0, 9241
Telemetry [candidate] (11.355 ms) : 0, 11355
Flare Poller [baseline] (3.484 ms) : 0, 3484
Flare Poller [candidate] (3.957 ms) : 0, 3957
section profiling
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.179 ms) : 0, 1179
BytebuddyAgent [baseline] (687.23 ms) : 0, 687230
BytebuddyAgent [candidate] (688.68 ms) : 0, 688680
AgentMeter [baseline] (9.042 ms) : 0, 9042
AgentMeter [candidate] (9.078 ms) : 0, 9078
GlobalTracer [baseline] (215.882 ms) : 0, 215882
GlobalTracer [candidate] (217.155 ms) : 0, 217155
AppSec [baseline] (32.429 ms) : 0, 32429
AppSec [candidate] (32.48 ms) : 0, 32480
Debugger [baseline] (66.326 ms) : 0, 66326
Debugger [candidate] (66.237 ms) : 0, 66237
Remote Config [baseline] (565.525 µs) : 0, 566
Remote Config [candidate] (560.473 µs) : 0, 560
Telemetry [baseline] (7.818 ms) : 0, 7818
Telemetry [candidate] (7.75 ms) : 0, 7750
Flare Poller [baseline] (3.55 ms) : 0, 3550
Flare Poller [candidate] (3.496 ms) : 0, 3496
ProfilingAgent [baseline] (94.419 ms) : 0, 94419
ProfilingAgent [candidate] (94.291 ms) : 0, 94291
Profiling [baseline] (94.994 ms) : 0, 94994
Profiling [candidate] (94.854 ms) : 0, 94854
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~3b91f88621, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055958
Total [baseline] (8.869 s) : 0, 8868587
Agent [candidate] (1.066 s) : 0, 1066068
Total [candidate] (8.857 s) : 0, 8857291
section iast
Agent [baseline] (1.227 s) : 0, 1227072
Total [baseline] (9.548 s) : 0, 9547968
Agent [candidate] (1.242 s) : 0, 1241588
Total [candidate] (9.56 s) : 0, 9559880
gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~3b91f88621, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.183 ms) : 0, 1183
crashtracking [candidate] (1.244 ms) : 0, 1244
BytebuddyAgent [baseline] (628.831 ms) : 0, 628831
BytebuddyAgent [candidate] (625.75 ms) : 0, 625750
AgentMeter [baseline] (29.431 ms) : 0, 29431
AgentMeter [candidate] (29.532 ms) : 0, 29532
GlobalTracer [baseline] (256.33 ms) : 0, 256330
GlobalTracer [candidate] (258.143 ms) : 0, 258143
AppSec [baseline] (31.787 ms) : 0, 31787
AppSec [candidate] (31.821 ms) : 0, 31821
Debugger [baseline] (59.442 ms) : 0, 59442
Debugger [candidate] (59.892 ms) : 0, 59892
Remote Config [baseline] (585.7 µs) : 0, 586
Remote Config [candidate] (593.566 µs) : 0, 594
Telemetry [baseline] (8.06 ms) : 0, 8060
Telemetry [candidate] (8.272 ms) : 0, 8272
Flare Poller [baseline] (4.302 ms) : 0, 4302
Flare Poller [candidate] (4.377 ms) : 0, 4377
section iast
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.227 ms) : 0, 1227
BytebuddyAgent [baseline] (796.63 ms) : 0, 796630
BytebuddyAgent [candidate] (795.819 ms) : 0, 795819
AgentMeter [baseline] (11.405 ms) : 0, 11405
AgentMeter [candidate] (11.756 ms) : 0, 11756
GlobalTracer [baseline] (247.371 ms) : 0, 247371
GlobalTracer [candidate] (249.49 ms) : 0, 249490
IAST [baseline] (25.297 ms) : 0, 25297
IAST [candidate] (25.801 ms) : 0, 25801
AppSec [baseline] (26.555 ms) : 0, 26555
AppSec [candidate] (27.028 ms) : 0, 27028
Debugger [baseline] (68.205 ms) : 0, 68205
Debugger [candidate] (64.528 ms) : 0, 64528
Remote Config [baseline] (516.353 µs) : 0, 516
Remote Config [candidate] (518.439 µs) : 0, 518
Telemetry [baseline] (10.297 ms) : 0, 10297
Telemetry [candidate] (13.787 ms) : 0, 13787
Flare Poller [baseline] (3.57 ms) : 0, 3570
Flare Poller [candidate] (4.71 ms) : 0, 4710
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 6 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~3b91f88621, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section baseline
no_agent (19.09 ms) : 18893, 19286
. : milestone, 19090,
appsec (18.547 ms) : 18358, 18736
. : milestone, 18547,
code_origins (17.629 ms) : 17455, 17804
. : milestone, 17629,
iast (17.991 ms) : 17810, 18171
. : milestone, 17991,
profiling (18.406 ms) : 18223, 18588
. : milestone, 18406,
tracing (17.63 ms) : 17454, 17806
. : milestone, 17630,
section candidate
no_agent (18.973 ms) : 18777, 19169
. : milestone, 18973,
appsec (18.694 ms) : 18505, 18884
. : milestone, 18694,
code_origins (17.648 ms) : 17472, 17824
. : milestone, 17648,
iast (18.082 ms) : 17903, 18261
. : milestone, 18082,
profiling (19.615 ms) : 19416, 19814
. : milestone, 19615,
tracing (17.645 ms) : 17467, 17823
. : milestone, 17645,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~3b91f88621, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section baseline
no_agent (1.206 ms) : 1194, 1218
. : milestone, 1206,
iast (3.199 ms) : 3157, 3240
. : milestone, 3199,
iast_FULL (5.767 ms) : 5710, 5824
. : milestone, 5767,
iast_GLOBAL (3.579 ms) : 3516, 3641
. : milestone, 3579,
profiling (1.922 ms) : 1907, 1938
. : milestone, 1922,
tracing (1.842 ms) : 1827, 1858
. : milestone, 1842,
section candidate
no_agent (1.185 ms) : 1172, 1198
. : milestone, 1185,
iast (3.194 ms) : 3150, 3238
. : milestone, 3194,
iast_FULL (6.01 ms) : 5950, 6071
. : milestone, 6010,
iast_GLOBAL (3.735 ms) : 3676, 3795
. : milestone, 3735,
profiling (2.141 ms) : 2122, 2161
. : milestone, 2141,
tracing (1.809 ms) : 1792, 1825
. : milestone, 1809,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~3b91f88621, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section baseline
no_agent (1.467 ms) : 1456, 1479
. : milestone, 1467,
appsec (3.799 ms) : 3578, 4021
. : milestone, 3799,
iast (2.248 ms) : 2179, 2317
. : milestone, 2248,
iast_GLOBAL (2.283 ms) : 2213, 2352
. : milestone, 2283,
profiling (2.094 ms) : 2039, 2149
. : milestone, 2094,
tracing (2.045 ms) : 1992, 2099
. : milestone, 2045,
section candidate
no_agent (1.467 ms) : 1456, 1479
. : milestone, 1467,
appsec (3.809 ms) : 3588, 4031
. : milestone, 3809,
iast (2.236 ms) : 2168, 2305
. : milestone, 2236,
iast_GLOBAL (2.291 ms) : 2222, 2361
. : milestone, 2291,
profiling (2.084 ms) : 2029, 2138
. : milestone, 2084,
tracing (2.06 ms) : 2006, 2113
. : milestone, 2060,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~3b91f88621, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section baseline
no_agent (14.956 s) : 14956000, 14956000
. : milestone, 14956000,
appsec (14.878 s) : 14878000, 14878000
. : milestone, 14878000,
iast (18.327 s) : 18327000, 18327000
. : milestone, 18327000,
iast_GLOBAL (17.965 s) : 17965000, 17965000
. : milestone, 17965000,
profiling (14.884 s) : 14884000, 14884000
. : milestone, 14884000,
tracing (14.818 s) : 14818000, 14818000
. : milestone, 14818000,
section candidate
no_agent (15.384 s) : 15384000, 15384000
. : milestone, 15384000,
appsec (14.595 s) : 14595000, 14595000
. : milestone, 14595000,
iast (18.42 s) : 18420000, 18420000
. : milestone, 18420000,
iast_GLOBAL (17.644 s) : 17644000, 17644000
. : milestone, 17644000,
profiling (15.289 s) : 15289000, 15289000
. : milestone, 15289000,
tracing (14.792 s) : 14792000, 14792000
. : milestone, 14792000,
|
`new int[probeCount][]` creates an array of null references. When the collector iterates `probeToLines[p]`, it NPEs on classes that have probes but no executable lines (no debug info, interfaces). Initialize each entry to `new int[0]` instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Skip classes without a SourceFile attribute in the collector to prevent NPE in the binary encoder (null is not a valid string in the coverage binary protocol) - Log instrumentation failures at debug level so they are visible when troubleshooting Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ClassProbeMappingBuilder previously ran JaCoCo's Analyzer N+1 times per class (once per probe + once for executable lines). For a class with 200 probes, that meant 201 full ASM parses of the same bytecode. The new implementation parses the class once using ClassProbesAdapter, builds a simplified instruction graph (ProbeNode with predecessor links), and walks predecessor chains to determine which lines each probe covers. ~200x faster for large classes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… scan On cache miss, try ClassLoader.getResourceAsStream() first (O(1) per class) before scanning all classpath jars/directories. Uses the system classloader (application classpath) first, then context classloader. CRC64 is verified after reading to ensure bytes match what was instrumented. Falls back to full classpath scan for any classes the classloader can't resolve. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace custom ScheduledExecutorService with a dedicated AgentTaskScheduler(CODE_COVERAGE) instance, following the same pattern as Profiler, Debugger, Remote Config, and Tracer Flare. This gives us proper daemon thread in the dd-trace-java thread group, null context classloader, uncaught exception handler, and shutdown hook handling for free. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The single-pass reimplementation had incorrect probe semantics for control flow: it unconditionally nulled currentInsn after every probe, but JaCoCo only does that for standalone visitProbe. This broke fall-through edges after probed conditional jumps and silently dropped later probed switch targets. Replace with a simpler O(probeCount) delegation to JaCoCo's Analyzer that is correct by construction, and add 33 unit tests covering branches, switches, loops, try-catch, nested conditions, and the specific bug scenarios. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes to match JaCoCo's MethodAnalyzer behavior: 1. addProbe() no longer unconditionally nulls currentInsn. This was breaking fall-through edges after probed conditional jumps and silently dropping later probed switch targets. 2. Only visitProbe() (standalone probes) sets currentInsn = null, matching JaCoCo where only visitProbe calls noSuccessor(). visitJumpInsnWithProbe, visitInsnWithProbe, and switch target probes correctly leave the chain intact. 3. Added one JaCoCo Analyzer pass (no probes) to obtain filtered executable lines (FinallyFilter, etc.). Probe-to-lines are intersected against these filtered lines so compiler-generated duplicates (e.g. finally block copies) are excluded. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Classes that are loaded and instrumented but never executed were invisible to the backend, making total coverage percentages inaccurate. Now the collector drains newly instrumented class names from the transformer each cycle and emits records with executable lines (covered lines empty) for classes that had no hits, so the backend has the full denominator. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The transformer now records CRC64 → WeakReference<ClassLoader> at transform time, when the JVM provides the exact defining classloader. ProbeMappingCache and the baseline reporting path use this recorded origin as the primary resolution strategy, falling back to system and context classloaders only if the defining loader is unavailable. This fixes resolution for classes loaded by custom classloaders (Spring Boot nested jars, OSGi, app-server modules, etc.) that are invisible to the system classloader and java.class.path. Other changes: - Removed recursive classpath scanning (scanDirectory, scanJar) and the explicitClasspath configuration — no longer needed - Removed permanent sentinel misses — unresolved classes are retried on subsequent collection cycles instead of being skipped forever - Transformer queues (classId, className) pairs instead of just names, so baseline resolution also uses origin-based lookup with CRC64 verification, eliminating the separate guess-based resolveClassBytes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The config key was parsed and exposed via getter but never read by the production coverage module, so the intended classpath fallback did not exist. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
since this is a large change and it's still in draft, let me ask @codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b91f88621
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| private static void openPackage( | ||
| Instrumentation inst, Class<?> classInPackage, ClassLoader targetLoader) throws Exception { | ||
| // module of the package to open (e.g. java.base for java.lang) | ||
| Object module = Class.class.getMethod("getModule").invoke(classInPackage); |
There was a problem hiding this comment.
CodeCoverageTransformer.openPackage unconditionally calls Class.getMethod("getModule") (and then uses java.lang.Module), but those APIs do not exist on Java 8. When dd.code.coverage.enabled is set on a Java 8 runtime, transformer initialization throws and Agent.maybeStartCodeCoverage falls back to disabling coverage, so the feature never starts on a still-supported JVM. Add a Java-version guard and bypass the JPMS-open path on pre-Java-9 runtimes.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.