Skip to content

test(appsec): avoid shell execution in taint tracking tests - #9657

Open
BridgeAR wants to merge 2 commits into
masterfrom
BridgeAR/2026-08-03-fix-taint-slice-flake
Open

test(appsec): avoid shell execution in taint tracking tests#9657
BridgeAR wants to merge 2 commits into
masterfrom
BridgeAR/2026-08-03-fix-taint-slice-flake

Conversation

@BridgeAR

@BridgeAR BridgeAR commented Aug 3, 2026

Copy link
Copy Markdown
Member

Summary

The taint propagation matrix armed a one-second trace deadline before each synchronous shell command. One Windows invocation blocked the event loop for 3.293 seconds, so the web span could not flush before the expectation expired.

The wrapped sink now rejects before process creation, preserving command injection detection without depending on OS process startup.

@BridgeAR
test(appsec): avoid shell execution in taint tracking tests
2478ca3 
The taint propagation matrix armed a one-second trace deadline before each synchronous shell command. One Windows invocation blocked the event loop for 3.293 seconds, so the web span could not flush before the expectation expired.
@dd-octo-sts

dd-octo-sts Bot commented Aug 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 7.96 MB
Deduped: 8.62 MB
No deduping: 8.62 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@pr-commenter

pr-commenter Bot commented Aug 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-07 13:34:52

Comparing candidate commit c17be3c in PR branch BridgeAR/2026-08-03-fix-taint-slice-flake with baseline commit eeab14c in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2307 metrics, 51 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-206.204ms; +210.821ms] or [-7.648%; +7.819%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-233.946ms; +231.232ms] or [-9.018%; +8.914%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-159931.493µs; +159173.393µs] or [-5.129%; +5.105%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-194.939ms; +189.600ms] or [-6.630%; +6.449%]

scenario:appsec-control-20

  • unstable execution_time [-85.336ms; +148.803ms] or [-5.192%; +9.054%]

scenario:appsec-control-24

  • unstable execution_time [-115802.812µs; +116565.678µs] or [-9.224%; +9.285%]

scenario:appsec-control-26

  • unstable execution_time [-128.756ms; +133.241ms] or [-10.247%; +10.604%]

scenario:appsec-iast-no-vulnerability-control-20

  • unstable execution_time [-15.958ms; +13.157ms] or [-6.161%; +5.079%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

  • unstable cpu_usage_percentage [-6.247%; +4.153%]
  • unstable execution_time [-20.165ms; +26.540ms] or [-7.480%; +9.844%]

scenario:child_process-shell-string-24

  • unstable cpu_usage_percentage [-8.816%; +2.887%]
  • unstable execution_time [-17.937ms; +30.669ms] or [-5.438%; +9.298%]
  • unstable throughput [-255713.661op/s; +186474.752op/s] or [-7.160%; +5.222%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-2636.860ms; +4194.057ms] or [-27.661%; +43.997%]
  • unstable execution_time [-2661.175ms; +4192.580ms] or [-25.955%; +40.891%]
  • unstable instructions [-23.6G instructions; +37.5G instructions] or [-29.596%; +47.113%]
  • unstable max_rss_usage [-8.451MB; +14.816MB] or [-5.343%; +9.367%]
  • unstable throughput [-821.531op/s; +524.731op/s] or [-25.395%; +16.220%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-2695.027ms; +2663.310ms] or [-30.626%; +30.266%]
  • unstable execution_time [-2720.145ms; +2713.465ms] or [-28.605%; +28.535%]
  • unstable instructions [-23348.9M instructions; +23224.4M instructions] or [-32.396%; +32.223%]
  • unstable max_rss_usage [-10364.032KB; +10391.232KB] or [-6.475%; +6.492%]
  • unstable throughput [-747.947op/s; +745.019op/s] or [-21.169%; +21.086%]

scenario:debugger-line-probe-with-snapshot-minimal-26

  • unstable cpu_user_time [-2339.484ms; +756.854ms] or [-24.502%; +7.927%]
  • unstable execution_time [-2351.373ms; +774.910ms] or [-22.892%; +7.544%]
  • unstable instructions [-20.6G instructions; +6.7G instructions] or [-25.865%; +8.460%]
  • unstable throughput [-155.304op/s; +458.608op/s] or [-4.805%; +14.190%]

scenario:debugger-line-probe-without-snapshot-24

  • unstable cpu_user_time [-1723.436ms; +667.445ms] or [-20.620%; +7.985%]
  • unstable execution_time [-1726.695ms; +668.898ms] or [-19.081%; +7.392%]
  • unstable instructions [-14.6G instructions; +5.6G instructions] or [-21.378%; +8.181%]
  • unstable throughput [-203.969op/s; +463.153op/s] or [-5.603%; +12.724%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-2683.066ms; +4206.579ms] or [-28.124%; +44.094%]
  • unstable execution_time [-2688.795ms; +4225.571ms] or [-26.231%; +41.223%]
  • unstable instructions [-23.6G instructions; +37.3G instructions] or [-29.680%; +46.880%]
  • unstable max_rss_usage [-8.918MB; +14.369MB] or [-5.651%; +9.105%]
  • unstable throughput [-825.270op/s; +531.139op/s] or [-25.492%; +16.407%]

scenario:dogstatsd-aggregated-26

  • unstable execution_time [-64.887ms; +57.790ms] or [-6.379%; +5.681%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-301.403ms; +374.002ms] or [-6.134%; +7.611%]
  • unstable execution_time [-304.537ms; +374.160ms] or [-6.111%; +7.508%]
  • unstable throughput [-133387.202op/s; +108702.945op/s] or [-7.916%; +6.451%]

scenario:log-with-debug-20

  • unstable max_rss_usage [-8.296MB; +5.166MB] or [-7.176%; +4.469%]

scenario:plugin-graphql-long-with-depth-and-collapse-off-20

  • unstable max_rss_usage [-20.741MB; +24.393MB] or [-5.318%; +6.254%]

scenario:plugin-graphql-long-with-depth-off-20

  • unstable max_rss_usage [-3.050MB; +11.597MB] or [-2.342%; +8.905%]

scenario:plugin-graphql-long-with-depth-off-26

  • unstable max_rss_usage [-31153.267KB; +31765.267KB] or [-16.469%; +16.793%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable cpu_user_time [-587.249ms; +579.847ms] or [-5.103%; +5.039%]
  • unstable execution_time [-606.048ms; +598.903ms] or [-5.159%; +5.098%]
  • unstable throughput [-3.500op/s; +3.564op/s] or [-5.111%; +5.203%]

scenario:plugin-pg-service-26

  • unstable cpu_usage_percentage [-6.154%; +4.248%]
  • unstable execution_time [-44.293ms; +88.492ms] or [-4.948%; +9.886%]
  • unstable throughput [-563868.302op/s; +277215.134op/s] or [-8.299%; +4.080%]

scenario:test-optimization-large-suite-20

  • unstable max_rss_usage [-5245.786KB; +3627.120KB] or [-6.598%; +4.562%]

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Aug 3, 2026 •
edited by datadog-datadog-prod-us1 Bot
Loading

Copy link
Copy Markdown

Tests

⚠️ Warnings

🧪 2 Tests failed in 1 job

All Green | all-green   GitHub Actions

Plugin aerospike with aerospike >=5.12.1 (5.12.1) without configuration client should instrument createIndex from client   View in Datadog 
AssertionError [ERR_ASSERTION]: Expected values to be partially and strictly deep-equal:
+ actual - expected

  {
+   duration: 4692871,
+   error: 0,
    meta: {
+     '_dd.base_service': 'test',
+     '_dd.git.commit.sha': '748383893964faeb6c6fbee4a89f76504b657535',
+     '_dd.git.repository_url': 'https://github.com/DataDog/dd-trace-js',
...
Plugin aerospike with aerospike >=5.12.1 (5.12.1) without configuration client should instrument query from client   View in Datadog 
AggregateError: Asserting traces failed. No result matched the expected one.
    at Timeout._onTimeout (/home/runner/work/dd-trace-js/dd-trace-js/packages/dd-trace/test/plugins/agent.js:388:15)
    at listOnTimeout (node:internal/timers:585:17)
    at process.processTimers (node:internal/timers:521:7) {
  errors: [
    AssertionError [ERR_ASSERTION]: Expected values to be partially and strictly deep-equal:
    + actual - expected
    
      {
    +   duration: 2264160,
...

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry View in Datadog

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 98.55% (+0.00%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c17be3c | Docs | Datadog PR Page | Give us feedback!

@codecov

codecov Bot commented Aug 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.55%. Comparing base (eeab14c) to head (c17be3c).

Additional details and impacted files 
@@           Coverage Diff            @@
##           master    #9657    +/-   ##
========================================
  Coverage   98.55%   98.55%            
========================================
  Files         966      966            
  Lines      138858   138858            
  Branches    12829    12058   -771     
========================================
  Hits       136845   136845            
  Misses       2013     2013
Flag Coverage Δ
aiguard 56.99% <ø> (-0.03%) ⬇️
aiguard-integration 55.90% <ø> (-0.01%) ⬇️
apm-bucket-0 57.24% <ø> (-0.03%) ⬇️
apm-bucket-1 63.39% <ø> (-0.02%) ⬇️
apm-bucket-2 62.24% <ø> (-0.03%) ⬇️
apm-bucket-3 59.82% <ø> (-0.03%) ⬇️
apm-capabilities-tracing 62.36% <ø> (ø)
apm-integrations-aerospike 56.30% <ø> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript 61.22% <ø> (-0.03%) ⬇️
apm-integrations-couchbase 56.72% <ø> (-0.03%) ⬇️
apm-integrations-http 61.93% <ø> (-0.02%) ⬇️
apm-integrations-kafkajs 61.74% <ø> (-0.03%) ⬇️
apm-integrations-next 59.42% <ø> (-0.03%) ⬇️
apm-integrations-prisma 58.53% <ø> (-0.02%) ⬇️
appsec 72.17% <ø> (-0.02%) ⬇️
appsec-express_fastify_graphql 69.48% <ø> (-0.02%) ⬇️
appsec-integration 50.17% <ø> (ø)
appsec-kafka_ldapjs_lodash 63.44% <ø> (-0.02%) ⬇️
appsec-mongodb-core_mongoose_mysql 66.92% <ø> (-0.02%) ⬇️
appsec-next 56.67% <ø> (-0.02%) ⬇️
appsec-node-serialize_passport_postgres 66.33% <ø> (-0.02%) ⬇️
appsec-sourcing_stripe_template 64.75% <ø> (-0.02%) ⬇️
debugger 64.29% <ø> (-0.04%) ⬇️
instrumentations-bucket-0 51.68% <ø> (-0.03%) ⬇️
instrumentations-bucket-1 59.41% <ø> (-0.03%) ⬇️
instrumentations-bucket-10 60.90% <ø> (-0.02%) ⬇️
instrumentations-bucket-11 61.63% <ø> (-0.02%) ⬇️
instrumentations-bucket-12 51.69% <ø> (-0.03%) ⬇️
instrumentations-bucket-13 52.43% <ø> (-0.03%) ⬇️
instrumentations-bucket-14 51.65% <ø> (-0.03%) ⬇️
instrumentations-bucket-2 53.14% <ø> (-0.03%) ⬇️
instrumentations-bucket-3 53.49% <ø> (-0.03%) ⬇️
instrumentations-bucket-4 58.78% <ø> (-0.03%) ⬇️
instrumentations-bucket-5 50.29% <ø> (-0.02%) ⬇️
instrumentations-bucket-6 60.52% <ø> (-0.03%) ⬇️
instrumentations-bucket-7 58.12% <ø> (-0.03%) ⬇️
instrumentations-bucket-8 59.09% <ø> (-0.03%) ⬇️
instrumentations-bucket-9 57.45% <ø> (-0.03%) ⬇️
instrumentations-instrumentation-couchbase 50.97% <ø> (-0.03%) ⬇️
instrumentations-integration-esbuild 34.24% <ø> (ø)
llmobs-ai_anthropic_bedrock 62.84% <ø> (-0.02%) ⬇️
llmobs-bucket-1 61.34% <ø> (-0.02%) ⬇️
llmobs-openai 61.81% <ø> (-0.02%) ⬇️
llmobs-openai-agents_vertex-ai 59.82% <ø> (-0.03%) ⬇️
llmobs-sdk 66.57% <ø> (-0.03%) ⬇️
master-coverage 98.55% <ø> (?)
openfeature 55.71% <ø> (ø)
openfeature-unit 53.22% <ø> (-0.03%) ⬇️
platform-core_esbuild_instrumentations-misc 41.22% <ø> (-0.02%) ⬇️
platform-integration 60.50% <ø> (ø)
platform-shimmer_unit-guardrails_webpack 38.88% <ø> (-0.02%) ⬇️
plugins-bucket-0 56.93% <ø> (-0.02%) ⬇️
plugins-bucket-1 54.05% <ø> (ø)
plugins-bucket-11 61.50% <ø> (-0.03%) ⬇️
plugins-bucket-17 61.32% <ø> (-0.03%) ⬇️
plugins-bucket-18 61.95% <ø> (-0.02%) ⬇️
plugins-bucket-19 61.34% <ø> (-0.03%) ⬇️
plugins-bucket-20 63.76% <ø> (-0.03%) ⬇️
plugins-bucket-4 58.32% <ø> (-0.03%) ⬇️
plugins-bullmq_cassandra_cookie 61.40% <ø> (-0.03%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 56.36% <ø> (-0.03%) ⬇️
plugins-fetch_fs_generic-pool 58.23% <ø> (-0.07%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 64.18% <ø> (-0.03%) ⬇️
plugins-hapi_hono_ioredis 59.92% <ø> (-0.03%) ⬇️
plugins-knex_langgraph_ldapjs 55.06% <ø> (-0.03%) ⬇️
plugins-light-my-request_limitd-client_lodash 58.38% <ø> (-0.04%) ⬇️
plugins-mariadb_memcached_mercurius 61.32% <ø> (-0.02%) ⬇️
plugins-mongodb_mongodb-core_mongoose 59.28% <ø> (-0.03%) ⬇️
plugins-multer_mysql_mysql2 58.86% <ø> (-0.03%) ⬇️
plugins-nats_node-serialize_opensearch 60.42% <ø> (-0.03%) ⬇️
plugins-passport-http_pino_postgres 58.60% <ø> (-0.03%) ⬇️
plugins-process_pug_redis 57.40% <ø> (-0.03%) ⬇️
plugins-undici_url_valkey 58.00% <ø> (-0.06%) ⬇️
plugins-vm_winston_ws 59.61% <ø> (-0.03%) ⬇️
profiling 61.54% <ø> (-0.03%) ⬇️
serverless-aws-sdk-aws-sdk 55.12% <ø> (-0.02%) ⬇️
serverless-aws-sdk-base-inject-field 50.92% <ø> (-0.03%) ⬇️
serverless-aws-sdk-bedrockruntime 54.65% <ø> (-0.02%) ⬇️
serverless-aws-sdk-client 56.22% <ø> (-0.03%) ⬇️
serverless-aws-sdk-dynamodb 55.50% <ø> (-0.02%) ⬇️
serverless-aws-sdk-eventbridge 49.71% <ø> (-0.02%) ⬇️
serverless-aws-sdk-kinesis 59.09% <ø> (-0.02%) ⬇️
serverless-aws-sdk-lambda 57.24% <ø> (-0.02%) ⬇️
serverless-aws-sdk-s3 55.59% <ø> (-0.02%) ⬇️
serverless-aws-sdk-serverless-peer-service 59.35% <ø> (-0.02%) ⬇️
serverless-aws-sdk-sns 59.90% <ø> (-0.02%) ⬇️
serverless-aws-sdk-sqs 60.32% <ø> (-0.02%) ⬇️
serverless-aws-sdk-stepfunctions 55.42% <ø> (-0.02%) ⬇️
serverless-aws-sdk-util 51.45% <ø> (-0.03%) ⬇️
serverless-bucket-0 54.13% <ø> (ø)
serverless-bucket-1 58.87% <ø> (-0.03%) ⬇️
test-optimization-cucumber 71.09% <ø> (-0.03%) ⬇️
test-optimization-cypress 64.92% <ø> (+0.10%) ⬆️
test-optimization-jest 72.64% <ø> (+0.13%) ⬆️
test-optimization-mocha 72.17% <ø> (+0.05%) ⬆️
test-optimization-playwright-playwright-atr 59.93% <ø> (+0.01%) ⬆️
test-optimization-playwright-playwright-efd 60.06% <ø> (+0.01%) ⬆️
test-optimization-playwright-playwright-final-status 60.23% <ø> (+0.01%) ⬆️
test-optimization-playwright-playwright-impacted-tests 59.76% <ø> (+0.16%) ⬆️
test-optimization-playwright-playwright-reporting 60.93% <ø> (-0.09%) ⬇️
test-optimization-playwright-playwright-test-management 60.75% <ø> (-0.09%) ⬇️
test-optimization-playwright-playwright-test-span 59.98% <ø> (-0.05%) ⬇️
test-optimization-selenium 59.12% <ø> (-0.13%) ⬇️
test-optimization-testopt 57.65% <ø> (+0.07%) ⬆️
test-optimization-vitest 73.38% <ø> (+0.05%) ⬆️
test-optimization-vitest-browser 58.99% <ø> (+0.01%) ⬆️
test-optimization-webdriverio 65.50% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BridgeAR
BridgeAR marked this pull request as ready for review August 7, 2026 13:22
@BridgeAR
BridgeAR requested a review from a team as a code owner August 7, 2026 13:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

appsec semver-patch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@BridgeAR