feat(otel): link Azure Durable Functions spans via W3C trace context

[chemystery09]

Summary

Adds OTel-only auto-instrumentation for Azure Functions and Durable Functions so HTTP, orchestration, activity, and entity spans link into a single distributed trace when:

1. DD_TRACE_OTEL_ENABLED=true (OTel API spans export through dd-trace)
2. DD_TRACE_AZURE_DURABLE_FUNCTIONS_ENABLED=false (native durable plugin disabled — OTel-only path)
3. Durable Functions distributed tracing V2 is enabled in the host (host.json)

This is the OTel-sourced span equivalent of the trace-linking work in #9394 / #9594 for the native azure-durable-functions plugin.

Scope: this PR adds trace linking via W3C context extraction. A related sampling visibility issue affects all OTel + Azure Durable V2 deployments (see below) and is documented here with a recommended app workaround until a library-side fix lands.

Motivation

Azure Durable Functions run as separate invocations connected via queues. Without extracting W3C traceparent/tracestate from InvocationContext.traceContext before creating spans, each invocation becomes an independent root trace in Datadog APM.

For apps using @opentelemetry/api with dd-trace as an export bridge (plugins: false), there was no built-in instrumentation to propagate durable trace context into OTel spans.

Changes

File	Purpose
helpers/azure-trace-context.js	Extract W3C context from InvocationContext.traceContext
helpers/otel-azure-enabled.js	Gate: OTel on + native durable plugin off
helpers/otel-azure-span.js	Shared OTel span helpers for Azure triggers
otel-azure-functions.js	HTTP triggers + app.generic orchestration triggers
otel-azure-durable-functions.js	app.orchestration, app.activity, app.entity
helpers/hooks.js	Load native Azure hooks first, then OTel hooks

Required app configuration

host.json (mandatory)

{
  "extensions": {
    "durableTask": {
      "tracing": {
        "distributedTracingEnabled": true,
        "version": "V2"
      }
    }
  }
}

Without this, InvocationContext.traceContext is empty on orchestration/activity invocations.

Environment variables

Setting	Value	Purpose
DD_TRACE_OTEL_ENABLED	true	Bridge OTel API → dd-trace export
DD_TRACE_AZURE_DURABLE_FUNCTIONS_ENABLED	false	Enable OTel-only durable instrumentation
DD_TRACE_SAMPLE_RATE	1	Recommended for local/dev (see sampling note)
_DD_APM_TRACING_AGENTLESS_ENABLED	true	Direct intake when no local Datadog agent (local dev only)

Typical OTel-only setup also uses ddtrace.init({ plugins: false }) so native dd-trace plugins do not duplicate spans.

Architecture

HTTP / Orchestration / Activity handler
  → read InvocationContext.traceContext (W3C traceparent/tracestate)
  → @opentelemetry/api span (@_dd.integration:otel)
  → dd-trace TracerProvider (export bridge)
  → Datadog intake

Behavior notes

• Orchestration spans are skipped when context.df.isReplaying is true (replay-safe).
• app.generic orchestration triggers (registered via @azure/functions) are also instrumented.
• Native azure-durable-functions plugin remains unchanged; OTel path activates only when that plugin is explicitly disabled.

Azure trace context sampling (known limitation)

This applies to any OTel-based Azure Durable Functions app using distributed tracing V2 — not specific to one sample app or local Azurite.

Azure's host forwards W3C traceparent with the sampled flag unset (flags=00). dd-trace honors that on extract, so spans created from propagated context are exported with _sampling_priority_v1: 0 and may be absent from APM Trace Explorer even when intake accepts them (HTTP 202). Setting DD_TRACE_SAMPLE_RATE=1 does not override priority already set from propagated context.

Recommended app workaround until library fix:

Register an OTel span processor that upgrades unsampled Azure-propagated spans to USER_KEEP via ddSpan._prioritySampler.setPriority(ddSpan, USER_KEEP).

Follow-up for dd-trace-js: re-apply tracer sampling inside helpers/azure-trace-context.js or helpers/otel-azure-span.js when starting spans from Azure-propagated context, so customers do not need app-level workarounds.

This does not affect the native azure-durable-functions dd-trace plugin path (#9394 / #9594).

CI / OTel-only design note

Disabling the native azure-durable-functions plugin (DD_TRACE_AZURE_DURABLE_FUNCTIONS_ENABLED=false) is an application/runtime configuration for OTel-only customers. It does not turn off dd-trace-js library tests or make tracing CI red by design.

What this PR does instead:

• Keeps native azure-functions / azure-durable-functions instrumentation registered (required for plugin-structure.spec.js and existing plugin coverage).
• Adds parallel OTel hooks that only create spans when DD_TRACE_OTEL_ENABLED=true and the native durable plugin is explicitly disabled.
• Avoids duplicate spans in OTel-only apps by gating on env flags, not by breaking library CI.

Initial tracing/instrumentations-misc failures on this branch were fixable implementation issues (hook require order tripping plugin-structure.spec.js, incomplete unit-test mocks), not an inherent consequence of the OTel-only path.

Test plan

• node node_modules/mocha/bin/mocha.js packages/datadog-instrumentations/test/helpers/azure-trace-context.spec.js packages/datadog-instrumentations/test/helpers/otel-azure.spec.js
• Azure Durable Functions sample app with host.json tracing V2 + OTel flags above
• Confirm one trace contains HTTP + orchestration + activity spans (single trace ID)
• Spans tagged @_dd.integration:otel
• Spans visible in APM (_sampling_priority_v1 >= 1, not rejected)
• Verify no duplicate spans when DD_TRACE_AZURE_DURABLE_FUNCTIONS_ENABLED=false and plugins: false

[dd-octo-sts]

Overall package size

Self size: 7.84 MB
Deduped: 8.5 MB
No deduping: 8.5 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

[datadog-datadog-prod-us1-2]

 

⚠️ Warnings

🚦 3 Pipeline jobs failed

All Green | all-green     

See error

Client does not have permission to upload reports for commits to the server: error 403 Forbidden.

🧪 1 Test failed

Plugin mongodb-core (core) with mongodb-core ~3.1.10 (3.1.10) using the server topology with dbmPropagationMode service DBM propagation should inject service mode as comment from with dbmPropagationMode service  

No matching trace received within 1000ms.

Error: No matching trace received within 1000ms.
    at Timeout._onTimeout (packages/dd-trace/test/plugins/agent.js:383:14)
    at listOnTimeout (node:internal/timers:581:17)
    at process.processTimers (node:internal/timers:519:7)

APM Integrations | mongodb-core     

🔄 Retry job. This looks flaky and may succeed on retry.

Dependency installation failed: Couldn't find versions for packages '@aws-sdk/credential-provider-sso' and '@aws-sdk/credential-provider-login'.

Instrumentation | instrumentations-misc     

See error

Error: Cannot find module '../../datadog-shimmer' during test execution.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 78.12%
• Overall Coverage: 97.09% (-1.43%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 57500dd | Docs | Datadog PR Page | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-08-04 18:34:49

Comparing candidate commit 568150b in PR branch ishara/otel-azure-durable-trace-propagation with baseline commit d70e6a0 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2324 metrics, 34 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

• unstable execution_time [-214.347ms; +225.029ms] or [-7.991%; +8.389%]

scenario:appsec-appsec-enabled-26

• unstable execution_time [-232445.372µs; +230858.839µs] or [-9.067%; +9.005%]

scenario:appsec-appsec-enabled-with-attacks-24

• unstable execution_time [-156397.418µs; +156118.784µs] or [-5.075%; +5.066%]

scenario:appsec-appsec-enabled-with-attacks-26

• unstable execution_time [-192143.654µs; +193324.587µs] or [-6.615%; +6.656%]

scenario:appsec-control-20

• unstable execution_time [-120.435ms; +125.045ms] or [-7.269%; +7.547%]

scenario:appsec-control-24

• unstable execution_time [-109632.220µs; +108106.154µs] or [-8.800%; +8.677%]

scenario:appsec-control-26

• unstable execution_time [-124.757ms; +130.471ms] or [-10.036%; +10.496%]

scenario:appsec-iast-no-vulnerability-control-20

• unstable execution_time [-15.327ms; +20.311ms] or [-5.893%; +7.809%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-20

• unstable execution_time [-27.036ms; +31.124ms] or [-4.921%; +5.666%]

scenario:debugger-line-probe-with-snapshot-default-24

• unstable cpu_user_time [-1753.846ms; +588.246ms] or [-21.067%; +7.066%]
• unstable execution_time [-1781.063ms; +587.916ms] or [-19.723%; +6.511%]
• unstable instructions [-15.1G instructions; +4.9G instructions] or [-22.251%; +7.183%]
• unstable throughput [-158.826op/s; +470.952op/s] or [-4.352%; +12.906%]

scenario:debugger-line-probe-without-snapshot-24

• unstable cpu_user_time [-2020.040ms; +3219.420ms] or [-24.409%; +38.902%]
• unstable execution_time [-2139.290ms; +3291.894ms] or [-23.805%; +36.631%]
• unstable instructions [-17.1G instructions; +27.3G instructions] or [-25.339%; +40.350%]
• unstable max_rss_usage [-8.137MB; +13.423MB] or [-5.190%; +8.562%]
• unstable throughput [-863.353op/s; +574.358op/s] or [-23.514%; +15.643%]

scenario:debugger-line-probe-without-snapshot-26

• unstable cpu_user_time [-2256.281ms; +732.393ms] or [-23.682%; +7.687%]
• unstable execution_time [-2277.709ms; +722.002ms] or [-22.203%; +7.038%]
• unstable instructions [-20.4G instructions; +6.7G instructions] or [-25.673%; +8.473%]
• unstable throughput [-143.547op/s; +454.332op/s] or [-4.446%; +14.072%]

scenario:dogstatsd-with-tags-20

• unstable cpu_user_time [-401.874ms; +155.756ms] or [-8.572%; +3.322%]
• unstable execution_time [-392.258ms; +154.802ms] or [-8.232%; +3.249%]
• unstable throughput [-56396.432op/s; +138874.246op/s] or [-3.207%; +7.897%]

scenario:plugin-claude-agent-sdk-compact-hook-indexed-26

• unstable cpu_usage_percentage [-6.764%; +3.713%]
• unstable execution_time [-3905.874µs; +5717.741µs] or [-4.586%; +6.713%]
• unstable throughput [-21008.747op/s; +16445.838op/s] or [-5.637%; +4.413%]

scenario:plugin-graphql-long-with-depth-and-collapse-off-20

• unstable max_rss_usage [-15.753MB; +23.841MB] or [-4.033%; +6.103%]

scenario:plugin-graphql-long-with-depth-off-26

• unstable max_rss_usage [-16.678MB; +46.912MB] or [-8.538%; +24.016%]

scenario:plugin-graphql-long-with-depth-on-max-20

• unstable cpu_user_time [-619.896ms; +598.959ms] or [-5.345%; +5.164%]
• unstable execution_time [-630.770ms; +611.596ms] or [-5.331%; +5.169%]
• unstable throughput [-3.536op/s; +3.677op/s] or [-5.200%; +5.406%]

scenario:test-optimization-large-suite-20

• unstable max_rss_usage [-3.404MB; +6.844MB] or [-4.283%; +8.612%]