Use canonical FFE fixtures by leoromanovsky · Pull Request #6901 · DataDog/system-tests

leoromanovsky · 2026-05-12T16:40:40Z

Motivation

Use the shared FFE fixture corpus in System Tests. This prevents copied fixtures from drifting and exercises one contract across supported SDKs.

The same migration is merged in Java and libdatadog. The Go migration is approved. Related migrations are open for Python, JavaScript, Ruby, and .NET.

Changes and Decisions

Add DataDog/ffe-system-test-data as a submodule under the parametric FFE tests.
Remove copied evaluator fixtures and load the canonical configuration and evaluation cases.
Assert canonical values and reasons for all 236 evaluation cases.
Fail selected FFE tests with an initialization instruction when the submodule is missing.
Initialize submodules in GitHub and GitLab test workflows.
Configure weekly Dependabot updates for the submodule.
Keep known SDK gaps in narrowly scoped manifest entries.

github-actions · 2026-05-12T16:41:15Z

CODEOWNERS have been resolved as:

.github/dependabot.yml                                                  @DataDog/system-tests-core
.gitmodules                                                             @DataDog/system-tests-core
.github/workflows/ci.yml                                                @DataDog/system-tests-core
.github/workflows/compute_libraries_and_scenarios.yml                   @DataDog/system-tests-core
.github/workflows/lint.yml                                              @DataDog/system-tests-core
.github/workflows/run-exotics.yml                                       @DataDog/system-tests-core
.github/workflows/run-parametric.yml                                    @DataDog/system-tests-core
.github/workflows/system-tests.yml                                      @DataDog/system-tests-core
.gitlab-ci.yml                                                          @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/ffe/README.md                                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_configuration_sources.py                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/ci/gitlab/templates.yml                                           @DataDog/system-tests-core
utils/mocked_backend/ffe.py                                             @DataDog/system-tests-core
tests/parametric/test_ffe/flags-v1.json                                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-boolean-false-assignment.json       @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-boolean-one-of-matches.json         @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-comparator-operator-flag.json       @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-disabled-flag.json                  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-empty-flag.json                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-empty-string-variation.json         @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-falsy-value-assignments.json        @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-flag-with-empty-string.json         @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-integer-flag.json                   @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-kill-switch-flag.json               @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-new-user-onboarding-flag.json       @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-no-allocations-flag.json            @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-null-operator-flag.json             @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-numeric-flag.json                   @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-numeric-one-of.json                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-of-7-empty-targeting-key.json       @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-regex-flag.json                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-start-and-end-date-flag.json        @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-flag-that-does-not-exist.json            @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-json-config-flag.json                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-no-allocations-flag.json                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-special-characters.json                  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-string-with-special-characters.json      @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-05-13T00:52:51Z

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 27 Pipeline jobs failed

Testing the test | System Tests (dotnet, dev) / parametric / parametric (2)

🔧 Fix in code. This looks caused by changes in this PR.

1 failed test. AssertionError: No RemoteConfig apply status found, got requests [] at tests/parametric/test_ffe/test_dynamic_evaluation.py:224.

🧪 1 Test failed · 🎯 related to the job error above

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_remote_config[library_env0, parametric-dotnet]

from system_tests_suite

AssertionError: No RemoteConfig apply status found, got requests [{&#39;headers&#39;: {&#39;Host&#39;: &#39;ddapm-test-agent-worker-gw7-5740354900026072187:8126&#39;, &#39;Datadog-Entity-ID&#39;: &#39;in-40901&#39;, &#39;Datadog-Meta-Lang&#39;: &#39;.NET&#39;, &#39;Datadog-Meta-Tracer-Version&#39;: &#39;3.51.0.0&#39;, &#39;x-datadog-tracing-enabled&#39;: &#39;false&#39;, &#39;Connection&#39;: &#39;close&#39;, &#39;Transfer-Encoding&#39;: &#39;chunked&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}, &#39;body&#39;: {&#39;client&#39;: {&#39;state&#39;: {&#39;root_version&#39;: 1, &#39;targets_version&#39;: 0, &#39;config_states&#39;: [], &#39;has_error&#39;: False, &#39;error&#39;: None, &#39;backend_client_state&#39;: None}, &#39;id&#39;: &#39;885eec58-5c98-482e-91f9-9228cbacaa66&#39;, &#39;products&#39;: [&#39;ASM_FEATURES&#39;, &#39;ASM_DD&#39;, &#39;FFE_FLAGS&#39;, &#39;APM_TRACING&#39;, &#39;AGENT_CONFIG&#39;, &#39;AGENT_TASK&#39;], &#39;is_tracer&#39;: True, &#39;client_tracer&#39;: {&#39;runtime_id&#39;: &#39;f3101a50-62e8-4674-8ec3-07075a851c5d&#39;, &#39;language&#39;: &#39;dotnet&#39;, &#39;tracer_version&#39;: &#39;3.51.0&#39;, &#39;service&#39;: &#39;apmtestapi&#39;, &#39;process_tags&#39;: [&#39;entrypoint.basedir:app&#39;, &#39;entrypoint.name:program&#39;, &#39;entrypoint.workdir:app&#39;, &#39;svc.auto:apmtestapi&#39;], &#39;extra_services&#39;: None, &#39;env&#39;: None, &#39;app_version&#39;: None, &#39;tags&#39;: [&#39;tracer_version:3.51.0&#39;, &#39;host_name:e3852b56b5d3&#39;, &#39;git.commit.sha:&#39;, &#39;git.repository_url:&#39;]}, &#39;capabilities&#39;: &#39;Y8CgCPf&#43;&#39;}, &#39;cached_target_files&#39;: []}, &#39;url&#39;: &#39;http://ddapm-test-agent-worker-gw7-5740354900026072187:8126/v0.7/config&#39;, &#39;method&#39;: &#39;POST&#39;}, {&#39;headers&#39;: {&#39;Host&#39;: &#39;ddapm-test-agent-worker-gw7-5740354900026072187:8126&#39;, &#39;Datadog-Entity-ID&#39;: &#39;in-40901&#39;, &#39;Datadog-Meta-Lang&#39;: &#39;.NET&#39;, &#39;Datadog-Meta-Tracer-Version&#39;: &#39;3.51.0.0&#39;, &#39;x-datadog-tracing-enabled&#39;: &#39;false&#39;, &#39;Connection&#39;: &#39;close&#39;, &#39;Transfer-Encoding&#39;: &#39;chunked&#39;, &#39;traceparent&#39;: &#39;00-0cebd5cc4e38641a17dd1d9fde81bf3a-289867dbf1d487de-00&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}, &#39;body&#39;: {&#39;client&#39;: {&#39;state&#39;: {&#39;root_version&#39;: 1, &#39;targets_version&#39;: 0, &#39;config_states&#39;: [], &#39;has_error&#39;: False, &#39;error&#39;: None, &#39;backend_client_state&#39;: None}, &#39;id&#39;: &#39;885eec58-5c98-482e-91f9-9228cbacaa66&#39;, &#39;products&#39;: [&#39;ASM_FEATURES&#39;, &#39;ASM_DD&#39;, &#39;FFE_FLAGS&#39;, &#39;APM_TRACING&#39;, &#39;AGENT_CONFIG&#39;, &#39;AGENT_TASK&#39;, &#39;LIVE_DEBUGGING_SYMBOL_DB&#39;], &#39;is_tracer&#39;: True, &#39;client_tracer&#39;: {&#39;runtime_id&#39;: &#39;f3101a50-62e8-4674-8ec3-07075a851c5d&#39;, &#39;language&#39;: &#39;dotnet&#39;, &#39;tracer_version&#39;: &#39;3.51.0&#39;, &#39;service&#39;: &#39;apmtestapi&#39;, &#39;process_tags&#39;: [&#39;entrypoint.basedir:app&#39;, &#39;entrypoint.name:program&#39;, &#39;entrypoint.workdir:app&#39;, &#39;svc.auto:apmtestapi&#39;], &#39;extra_services&#39;: None, &#39;env&#39;: None, &#39;app_version&#39;: None, &#39;tags&#39;: [&#39;tracer_version:3.51.0&#39;, &#39;host_name:e3852b56b5d3&#39;, &#39;git.commit.sha:&#39;, &#39;git.repository_url:&#39;]}, &#39;capabilities&#39;: &#39;Y8CgCPf&#43;&#39;}, &#39;cached_target_files&#39;: []}, &#39;url&#39;: &#39;http://ddapm-test-agent-worker-gw7-5740354900026072187:8126/v0.7/config&#39;, &#39;method&#39;: &#39;POST&#39;}, {&#39;headers&#39;: {&#39;Host&#39;: &#39;ddapm-test-agent-worker-gw7-5740354900026072187:8126&#39;, &#39;Datadog-Entity-ID&#39;: &#39;in-40901&#39;, &#39;Datadog-Meta-Lang&#39;: &#39;.NET&#39;, &#39;Datadog-Meta-Tracer-Version&#39;: &#39;3.51.0.0&#39;, &#39;x-datadog-tracing-enabled&#39;: &#39;false&#39;, &#39;Connection&#39;: &#39;close&#39;, &#39;Transfer-Encoding&#39;: &#39;chunked&#39;, &#39;traceparent&#39;: &#39;00-f8cf3da9dad7e1e5bde65f058607a40e-b0b7cedafb3899bb-00&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}, &#39;body&#39;: {&#39;client&#39;: {&#39;state&#39;: {&#39;root_version&#39;: 1, &#39;targets_version&#39;: 0, &#39;config_states&#39;: [], &#39;has_error&#39;: False, &#39;error&#39;: None, &#39;backend_client_state&#39;: None}, &#39;id&#39;: &#39;885eec58-5c98-482e-91f9-9228cbacaa66&#39;, &#39;products&#39;: [&#39;ASM_FEATURES&#39;, &#39;ASM_DD&#39;, &#39;FFE_FLAGS&#39;, &#39;APM_TRACING&#39;, &#39;AGENT_CONFIG&#39;, &#39;AGENT_TASK&#39;, &#39;LIVE_DEBUGGING_SYMBOL_DB&#39;], &#39;is_tracer&#39;: True, &#39;client_tracer&#39;: {&#39;runtime_id&#39;: &#39;f3101a50-62e8-4674-8ec3-07075a851c5d&#39;, &#39;language&#39;: &#39;dotnet&#39;, &#39;tracer_version&#39;: &#39;3.51.0&#39;, &#39;service&#39;: &#39;apmtestapi&#39;, &#39;process_tags&#39;: [&#39;entrypoint.basedir:app&#39;, &#39;entrypoint.name:program&#39;, &#39;entrypoint.workdir:app&#39;, &#39;svc.auto:apmtestapi&#39;], &#39;extra_services&#39;: None, &#39;env&#39;: None, &#39;app_version&#39;: None, &#39;tags&#39;: [&#39;tracer_version:3.51.0&#39;, &#39;host_name:e3852b56b5d3&#39;, &#39;git.commit.sha:&#39;, &#39;git.repository_url:&#39;]}, &#39;capabilities&#39;: &#39;Y8CgCPf&#43;&#39;}, &#39;cached_target_files&#39;: []}, &#39;url&#39;: &#39;http://ddapm-test-agent-worker-gw7-5740354900026072187:8126/v0.7/config&#39;, &#39;method&#39;: &#39;POST&#39;}, {&#39;headers&#39;: {&#39;Host&#39;: &#39;ddapm-test-agent-worker-gw7-5740354900026072187:8126&#39;, &#39;Datadog-Entity-ID&#39;: &#39;in-40901&#39;, &#39;Datadog-Meta-Lang&#39;: &#39;.NET&#39;, &#39;Datadog-Meta-Tracer-Version&#39;: &#39;3.51.0.0&#39;, &#39;x-datadog-tracing-enabled&#39;: &#39;false&#39;, &#39;Connection&#39;: &#39;close&#39;, &#39;Transfer-Encoding&#39;: &#39;chunked&#39;, &#39;traceparent&#39;: &#39;00-99d122a3af7890ac7d55044fa2b99c23-40072de5f1c9cbb9-00&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}, &#39;body&#39;: {&#39;client&#39;: {&#39;state&#39;: {&#39;root_version&#39;: 1, &#39;targets_version&#39;: 0, &#39;config_states&#39;: [], &#39;has_error&#39;: False, &#39;error&#39;: None, &#39;backend_client_state&#39;: None}, &#39;id&#39;: &#39;885eec58-5c98-482e-91f9-9228cbacaa66&#39;, &#39;products&#39;: [&#39;ASM_FEATURES&#39;, &#39;ASM_DD&#39;, &#39;FFE_FLAGS&#39;, &#39;APM_TRACING&#39;, &#39;AGENT_CONFIG&#39;, &#39;AGENT_TASK&#39;, &#39;LIVE_DEBUGGING_SYMBOL_DB&#39;], &#39;is_tracer&#39;: True, &#39;client_tracer&#39;: {&#39;runtime_id&#39;: &#39;f3101a50-62e8-4674-8ec3-07075a851c5d&#39;, &#39;language&#39;: &#39;dotnet&#39;, &#39;tracer_version&#39;: &#39;3.51.0&#39;, &#39;service&#39;: &#39;apmtestapi&#39;, &#39;process_tags&#39;: [&#39;entrypoint.basedir:app&#39;, &#39;entrypoint.name:program&#39;, &#39;entrypoint.workdir:app&#39;, &#39;svc.auto:apmtestapi&#39;], &#39;extra_services&#39;: None, &#39;env&#39;: None, &#39;app_version&#39;: None, &#39;tags&#39;: [&#39;

Testing the test | System Tests (nodejs, dev) / parametric / parametric (2)

🔧 Fix in code. This looks caused by changes in this PR.

Test run failed. Assertion Error: Test case 0 in test-case-of-7-empty-targeting-key.json failed: flag='empty-targeting-key-flag', targetingKey='', expected reason=STATIC, actual reason=DEFAULT

🧪 25 Tests failed · 🎯 related to the job error above

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-of-7-empty-targeting-key.json-library_env0, parametric-nodejs]

from system_tests_suite

AssertionError: Test case 0 in test-case-of-7-empty-targeting-key.json failed: flag=&#39;empty-targeting-key-flag&#39;, targetingKey=&#39;&#39;, expected reason=STATIC, actual reason=DEFAULT
assert &#39;DEFAULT&#39; == &#39;STATIC&#39;
  - STATIC
  &#43; DEFAULT

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f6d637b7e30&gt;
test_case_file = &#39;test-case-of-7-empty-targeting-key.json&#39;
ufc_fixture_data = {&#39;createdAt&#39;: &#39;2024-04-17T19:40:53.716Z&#39;, &#39;environment&#39;: {&#39;name&#39;: &#39;Test&#39;, &#39;unknownEnvironmentField&#39;: &#39;ignored&#39;}, &#39;flag...ations&#39;: [], &#39;enabled&#39;: False, &#39;key&#39;: &#39;disabled_flag&#39;, &#39;variationType&#39;: &#39;INTEGER&#39;, ...}, ...}, &#39;format&#39;: &#39;SERVER&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f6d623ebad0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f6d6234b770&gt;
...

View all failed tests

Testing the test | System Tests (php, dev) / parametric / parametric (2)

🔧 Fix in code. This looks caused by changes in this PR.

6 failed tests. Assertion Error: failed evaluations for 'test-case-empty-flag.json', 'test-case-invalid-shard-bounds-isolation.json', 'test-case-disabled-flag.json', 'test-case-malformed-flag-isolation.json', 'test-case-missing-split-shards-isolation.json', and 'test-case-unknown-operator-isolation.json'.

🧪 6 Tests failed · 🎯 related to the job error above

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-disabled-flag.json-library_env0, parametric-php]

from system_tests_suite

AssertionError: Test case 0 in test-case-disabled-flag.json failed: flag=&#39;disabled_flag&#39;, targetingKey=&#39;alice&#39;, expected reason=DISABLED, actual reason=ERROR
assert &#39;ERROR&#39; == &#39;DISABLED&#39;
  - DISABLED
  &#43; ERROR

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7fed0d1e3f20&gt;
test_case_file = &#39;test-case-disabled-flag.json&#39;
ufc_fixture_data = {&#39;createdAt&#39;: &#39;2024-04-17T19:40:53.716Z&#39;, &#39;environment&#39;: {&#39;name&#39;: &#39;Test&#39;, &#39;unknownEnvironmentField&#39;: &#39;ignored&#39;}, &#39;flag...ations&#39;: [], &#39;enabled&#39;: False, &#39;key&#39;: &#39;disabled_flag&#39;, &#39;variationType&#39;: &#39;INTEGER&#39;, ...}, ...}, &#39;format&#39;: &#39;SERVER&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fecd9aaa330&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fecda1a03b0&gt;
...

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-empty-flag.json-library_env0, parametric-php]

from system_tests_suite

AssertionError: Test case 0 in test-case-empty-flag.json failed: flag=&#39;empty_flag&#39;, targetingKey=&#39;alice&#39;, expected reason=DEFAULT, actual reason=ERROR
assert &#39;ERROR&#39; == &#39;DEFAULT&#39;
  - DEFAULT
  &#43; ERROR

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f2b377dc080&gt;
test_case_file = &#39;test-case-empty-flag.json&#39;
ufc_fixture_data = {&#39;createdAt&#39;: &#39;2024-04-17T19:40:53.716Z&#39;, &#39;environment&#39;: {&#39;name&#39;: &#39;Test&#39;, &#39;unknownEnvironmentField&#39;: &#39;ignored&#39;}, &#39;flag...ations&#39;: [], &#39;enabled&#39;: False, &#39;key&#39;: &#39;disabled_flag&#39;, &#39;variationType&#39;: &#39;INTEGER&#39;, ...}, ...}, &#39;format&#39;: &#39;SERVER&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f2b07556780&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f2b07555b50&gt;
...

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-invalid-shard-bounds-isolation.json-library_env0, parametric-php]

from system_tests_suite

AssertionError: Test case 0 in test-case-invalid-shard-bounds-isolation.json failed: flag=&#39;overflow-total-shards-flag&#39;, targetingKey=&#39;alice&#39;, expected reason=DEFAULT, actual reason=ERROR
assert &#39;ERROR&#39; == &#39;DEFAULT&#39;
  - DEFAULT
  &#43; ERROR

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f73bd963d10&gt;
test_case_file = &#39;test-case-invalid-shard-bounds-isolation.json&#39;
ufc_fixture_data = {&#39;createdAt&#39;: &#39;2024-04-17T19:40:53.716Z&#39;, &#39;environment&#39;: {&#39;name&#39;: &#39;Test&#39;, &#39;unknownEnvironmentField&#39;: &#39;ignored&#39;}, &#39;flag...ations&#39;: [], &#39;enabled&#39;: False, &#39;key&#39;: &#39;disabled_flag&#39;, &#39;variationType&#39;: &#39;INTEGER&#39;, ...}, ...}, &#39;format&#39;: &#39;SERVER&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f73bcacbad0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f73bcd7a7b0&gt;
...

View all failed tests

View all 27 failed jobs.

📋 Copy prompt for your agent

CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: leo.romanovsky/ffe-canonical-fixtures-20260512

Testing the test | System Tests (dotnet, dev) / parametric / parametric (2)
Commit: 946258aa92c9cadecbbdba943ca9b0c81f309327
Error (code / test):
1 failed test. AssertionError: No RemoteConfig apply status found, got requests [] at tests/parametric/test_ffe/test_dynamic_evaluation.py:224.
CI job: https://github.com/DataDog/system-tests/actions/runs/30771273562/job/91567499463

Testing the test | System Tests (nodejs, dev) / parametric / parametric (2)
Commit: 946258aa92c9cadecbbdba943ca9b0c81f309327
Error (code / test):
4 failed tests. Assertion Error: Test case 0 in test-case-of-7-empty-targeting-key.json failed: flag='empty-targeting-key-flag', targetingKey='', expected reason=STATIC, actual reason=DEFAULT
CI job: https://github.com/DataDog/system-tests/actions/runs/30771273562/job/91567499292

Testing the test | System Tests (php, dev) / parametric / parametric (2)
Commit: 946258aa92c9cadecbbdba943ca9b0c81f309327
Error (code / test):
6 failed tests. Assertion Error: failed evaluations for 'test-case-empty-flag.json', 'test-case-invalid-shard-bounds-isolation.json', 'test-case-disabled-flag.json', 'test-case-malformed-flag-isolation.json', 'test-case-missing-split-shards-isolation.json', and 'test-case-unknown-operator-isolation.json'.
CI job: https://github.com/DataDog/system-tests/actions/runs/30771273562/job/91567499588

Plus 6 more failing jobs not shown here.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 4 jobs - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 1b018b5 | Docs | Datadog PR Page | Give us feedback!}

…nonical-fixtures-20260512 # Conflicts: # .gitlab-ci.yml # tests/parametric/test_ffe/test_configuration_sources.py

leoromanovsky added 5 commits July 13, 2026 13:15

Use canonical FFE fixtures

690781e

Fix FFE fixture submodule checkout

255cd7a

Avoid FFE fixture collection failures

e8ba1b7

Mark main FFE null targeting key gaps

96baf39

Integrate canonical FFE fixtures into CI

9be6f96

leoromanovsky force-pushed the leo.romanovsky/ffe-canonical-fixtures-20260512 branch from cf3498a to 9be6f96 Compare July 13, 2026 17:45

leoromanovsky added 3 commits July 13, 2026 17:39

Gate new FFE cases for .NET

67b2399

Merge remote-tracking branch 'origin/main' into leo.romanovsky/ffe-ca…

9c6c866

…nonical-fixtures-20260512 # Conflicts: # .gitlab-ci.yml # tests/parametric/test_ffe/test_configuration_sources.py

Merge branch 'main' into leo.romanovsky/ffe-canonical-fixtures-20260512

1b018b5

leoromanovsky mentioned this pull request Aug 14, 2026

test(ffe): use checked-in canonical fixture snapshot #7511

Draft

Navigation Menu

Uh oh!

Use canonical FFE fixtures - #6901

Use canonical FFE fixtures#6901
leoromanovsky wants to merge 8 commits into
mainfrom
leo.romanovsky/ffe-canonical-fixtures-20260512

leoromanovsky commented May 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 12, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented May 13, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

leoromanovsky commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes and Decisions

Uh oh!

github-actions Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

leoromanovsky commented May 12, 2026 •

edited

Loading

github-actions Bot commented May 12, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented May 13, 2026 •

edited

Loading