-
Notifications
You must be signed in to change notification settings - Fork 330
Description
Note
The below is an LLM-generated report from the full application log.
The issue is real, but I am not sure how actionable it is, since the bug may be either in DD lib, byte-buddy or JDK.
I checked for similar issues in ByteBuddy, dd-trace-java, OpenTelemetry Java, OpenJDK, Corretto 25, did not find anything.
I am not attaching anything since I saw in the logs it already uploaded the full log.
That log should have even more than I see in my log.
"[dd.trace 2026-03-23 14:02:38:544 +0000] [OkHttp http://localhost:8126/...] INFO datadog.crashtracking.CrashUploader - Successfully uploaded the crash to http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry, code = 200 ""OK"""
Error file /app/hs_err_pid531.log was uploaded successfully
Tracer Version(s)
1.60.2
Java Version(s)
25.0.2+10
JVM Vendor
Amazon Corretto
Bug Report
Our custody service crashed with a SIGSEGV after ~12h of uptime. The crash was auto-uploaded via the Datadog crash tracker (timestamp: 2026-03-23 14:02:37 UTC, runtime-id: 087dee7d-428c-4557-83a6-ff3514b6adc0, env: stg).
The crash occurs in the Exception Debugger's retransformation path when retransforming a Java Record class.
Crash thread: dd-task-scheduler
*Native stack trace:
V [libjvm.so+0x74c348] ConstantPool::klass_name_at(int) const+0x8
V [libjvm.so+0xc23b88] VM_RedefineClasses::compare_and_normalize_class_versions(InstanceKlass*, InstanceKlass*)+0x2e8
V [libjvm.so+0xc27192] VM_RedefineClasses::load_new_class_versions() [clone .part.0]+0x2c2
V [libjvm.so+0xc27eb4] VM_RedefineClasses::doit_prologue()+0x1c4
V [libjvm.so+0x1191350] VMThread::execute(VM_Operation*)+0x40
V [libjvm.so+0xbf7538] JvmtiEnv::RetransformClasses(int, _jclass* const*)+0x4c8
V [libjvm.so+0xba1549] jvmti_RetransformClasses+0xf9
C [libjavaProfiler-dd-tmp9337984804387176169.so+0x595e0]
C [libinstrument.so+0x5dc6] retransformClasses+0x1b6
Java stack trace:
j sun.instrument.InstrumentationImpl.retransformClasses0(J[Ljava/lang/Class;)V+0
j sun.instrument.InstrumentationImpl.retransformClasses([Ljava/lang/Class;)V+35
j com.datadog.debugger.agent.ConfigurationUpdater.retransformIndividualClasses(Ljava/util/List;)V+53
j com.datadog.debugger.agent.ConfigurationUpdater.retransformClasses(Ljava/util/List;)V+15
j com.datadog.debugger.agent.ConfigurationUpdater.handleProbesChanges(...)V+58
j com.datadog.debugger.agent.ConfigurationUpdater.applyNewConfiguration(...)V+70
j com.datadog.debugger.agent.ConfigurationUpdater.accept(...)V+32
j com.datadog.debugger.exception.AbstractExceptionDebugger.applyExceptionConfiguration(...)V+14
j com.datadog.debugger.exception.AbstractExceptionDebugger.lambda$handleException$0(...)V+2
The JVM holds two versions of the same Record class during retransformation. The old version (state: fully_initialized, flags: is_being_redefined has_been_redefined) has correct field metadata, but the new version (state: allocated) produced by the transformation has corrupted field layout — field names and types are mangled. This suggests the bytecode transformation produced an invalid constant pool for the Record class.
Signal: SIGSEGV (0xb), SEGV_ACCERR at si_addr: 0x00007ff419fe9990
JVM command line:
-XX:FlightRecorderOptions=stackdepth=256 -Xms448m -Xmx2944m -XX:MetaspaceSize=256m -XX:-OmitStackTraceInFastThrow -javaagent:/app/dd-java-agent.jar /app/app.jar
Environment: Amazon Linux 2023, ECS Fargate, 2 cores, 4GB RAM
The full hs_err_pid531.log was successfully uploaded via the crash tracker. The class being retransformed is a simple Java Record with a single LocalDate field.
Expected Behavior
The Exception Debugger should safely retransform Record classes on JDK 25 without causing a JVM crash. If the transformation produces bytecode incompatible with the JVM version, it should fail gracefully with a Java-level exception rather than a native SIGSEGV.
Reproduction Code
No specific reproduction code — this occurred after ~12h of uptime in production (staging environment) when the Exception Debugger attempted to retransform a Java Record class. The full hs_err crash log was auto-uploaded via the Datadog crash tracker (see details in the bug report).