test(openfeature): cover isolated malformed flags by leoromanovsky · Pull Request #6179 · DataDog/dd-trace-rb

leoromanovsky · 2026-08-11T16:36:23Z

Motivation

The updated canonical FFE corpus verifies malformed-flag isolation and exposes shared libdatadog behavior that Ruby should consume without duplicating evaluator logic.

Changes

integrate the canonical fixture submodule and pin it to ea8b5cc5ce335109f11f3efbc5fd608f98a3ca54
preserve native ERROR/PARSE_ERROR details while applying the caller default value
remove the Ruby-side malformed-configuration workaround
assert canonical reasons, variants, and error codes

Decisions

keep malformed-flag isolation and SemVer precedence in fix(ffe): report rejected flags as parse errors libdatadog#2339
do not source-pin libdatadog: dd-trace-rb consumes packaged platform artifacts from libdatadog-rb
bump datadog.gemspec and ext/libdatadog_extconf_helpers.rb together only after the fixed binary gem is published
keep the separate regex-fixture proposal in ffe-system-test-data#21 out of scope

Validation

Ruby 3.2 native extension compilation
contract spec passed on latest and minimum OpenFeature appraisals
StandardRB, targeted Steep, and git diff --check
full OpenFeature suite: 499 examples; 13 dependency-gated failures on released libdatadog 40.0.0.1.0 (12 malformed-flag results and one SemVer build-metadata result)

Depends on DataDog/libdatadog#2339 and its follow-up binary gem publication.

Bring the canonical evaluator fixtures onto current master before testing the PR #21 regex additions. Environment: Datadog workspace

dd-octo-sts · 2026-08-11T16:36:37Z

👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description.

If changes need to be present in CHANGELOG.md you can state it this way

**Change log entry**

Yes. A brief summary to be placed into the CHANGELOG.md

(possible answers Yes/Yep/Yeah)

Or you can opt out like that

**Change log entry**

None.

(possible answers No/Nope/None)

^{Visited at: 2026-08-11 16:43:26 UTC}

dd-octo-sts · 2026-08-11T16:36:57Z

Typing analysis

Note: Ignored files are excluded from the next sections.

Untyped methods

This PR introduces 1 partially typed method, and clears 2 partially typed methods. It increases the percentage of typed methods from 69.52% to 69.53% (+0.01%).

Partially typed methods (+1-2)

❌ Introduced:

sig/datadog/open_feature/native_evaluator.rbs:8
└── def get_assignment: (
        ::String flag_key,
        default_value: untyped,
        context: ::OpenFeature::SDK::EvaluationContext::fields_t,
        expected_type: ::Symbol
      ) -> Core::FeatureFlags::ResolutionDetails

✅ Cleared:

sig/datadog/open_feature/native_evaluator.rbs:10
└── def get_assignment: (
        ::String flag_key,
        default_value: untyped,
        context: ::OpenFeature::SDK::EvaluationContext::fields_t,
        expected_type: ::Symbol
      ) -> (Core::FeatureFlags::ResolutionDetails | ResolutionDetails)
sig/datadog/open_feature/native_evaluator.rbs:21
└── def invalid_flag_configuration_error: (untyped default_value) -> ResolutionDetails

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

datadog-datadog-us1-prod · 2026-08-11T16:46:02Z

⚠️ Warnings

🚦 7 Pipeline jobs failed

System Tests | test / End-to-end #4 / rails72 4

🔧 Fix in code. This looks caused by changes in this PR.

2 failed tests. AssertionError: Expected reason 'error' for parse error, got tags: ['env:system-tests', 'feature_flag.key:eval-metric-parse-error-flag', 'feature_flag.result.reason:default', ...'.

🧪 2 Tests failed · 🎯 related to the job error above

🎯

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex.test_ffe_eval_metric_parse_error_invalid_regex[rails72]

from system_tests_suite

AssertionError: Expected reason &#39;error&#39; for parse error, got tags: [&#39;env:system-tests&#39;, &#39;feature_flag.key:eval-metric-parse-error-flag&#39;, &#39;feature_flag.result.reason:default&#39;, &#39;feature_flag.result.variant:n/a&#39;, &#39;instrumentation_scope:ddtrace.openfeature&#39;, &#39;instrumentation_scope_version:n/a&#39;, &#39;service:weblog&#39;, &#39;version:1.0.0&#39;]
assert &#39;default&#39; == &#39;error&#39;
  - error
  &#43; default

self = &lt;tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex object at 0x7f292ae59580&gt;

    def test_ffe_eval_metric_parse_error_invalid_regex(self):
        &#34;&#34;&#34;Test that invalid regex produces error.type:parse_error.&#34;&#34;&#34;
        assert self.r.status_code == 200, f&#34;Flag evaluation request failed: {self.r.text}&#34;
...

🎯

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch.test_ffe_eval_metric_parse_error_variant_type_mismatch[rails72]

from system_tests_suite

AssertionError: Expected reason &#39;error&#39; for variant type mismatch, got tags: [&#39;env:system-tests&#39;, &#39;feature_flag.key:eval-variant-type-mismatch-flag&#39;, &#39;feature_flag.result.reason:default&#39;, &#39;feature_flag.result.variant:n/a&#39;, &#39;instrumentation_scope:ddtrace.openfeature&#39;, &#39;instrumentation_scope_version:n/a&#39;, &#39;service:weblog&#39;, &#39;version:1.0.0&#39;]
assert &#39;default&#39; == &#39;error&#39;
  - error
  &#43; default

self = &lt;tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch object at 0x7f292af65b80&gt;

    def test_ffe_eval_metric_parse_error_variant_type_mismatch(self):
        &#34;&#34;&#34;Test that variant type mismatch produces error.type:parse_error.&#34;&#34;&#34;
        assert self.r.status_code == 200, f&#34;Flag evaluation request failed: {self.r.text}&#34;
...

Unit Tests | Ruby 3.1 / build & test (standard) [6]

🔧 Fix in code. This looks caused by changes in this PR.

ArgumentError: wrong number of arguments (given 1, expected 0) in /lib/datadog/open_feature/native_evaluator.rb:14

Unit Tests | Ruby 3.2 / build & test (standard) [6]

🔧 Fix in code. This looks caused by changes in this PR.

13 failed tests related to encoding mismatch. Expected UTF-8 but got ASCII-8BIT in multiple cases.

View all 7 failed jobs.

📋 Copy prompt for your agent

CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: agent/isolate-malformed-flags

System Tests | test / End-to-end #4 / rails72 4
Commit: 37435727f3786a9a18ffd47412ad3a570d38b9f4
Error (code / test):
2 failed tests. AssertionError: Expected reason 'error' for parse error, got tags: ['env:system-tests', 'feature_flag.key:eval-metric-parse-error-flag', 'feature_flag.result.reason:default', ...'.
CI job: https://github.com/DataDog/dd-trace-rb/actions/runs/31514714464/job/93858126537

Unit Tests | Ruby 3.1 / build & test (standard) [6]
Commit: 37435727f3786a9a18ffd47412ad3a570d38b9f4
Error (code / build):
ArgumentError: wrong number of arguments (given 1, expected 0) in /lib/datadog/open_feature/native_evaluator.rb:14
CI job: https://github.com/DataDog/dd-trace-rb/actions/runs/31514714047/job/93857338545

Unit Tests | Ruby 3.2 / build & test (standard) [6]
Commit: 37435727f3786a9a18ffd47412ad3a570d38b9f4
Error (code / test):
13 failed tests related to encoding mismatch. Expected UTF-8 but got ASCII-8BIT in multiple cases.
CI job: https://github.com/DataDog/dd-trace-rb/actions/runs/31514714047/job/93857265096

Plus 3 more failing jobs not shown here.

❄️ 20 New flaky tests detected

Datadog::OpenFeature::NativeEvaluator canonical FFE fixtures evaluates test-case-semver-comparison-flag.json[0] from rspec

wrong number of arguments (given 1, expected 0)

Failure/Error: @configuration = Core::FeatureFlags::Configuration.new(configuration)

ArgumentError:
  wrong number of arguments (given 1, expected 0)
./lib/datadog/open_feature/native_evaluator.rb:14:in \`initialize&#39;
./lib/datadog/open_feature/native_evaluator.rb:14:in \`new&#39;
./lib/datadog/open_feature/native_evaluator.rb:14:in \`initialize&#39;
./spec/datadog/open_feature/native_evaluator_spec.rb:94:in \`new&#39;
...

New test introduced in this PR is flaky.

Datadog::OpenFeature::NativeEvaluator canonical FFE fixtures evaluates test-case-semver-comparison-flag.json[10] from rspec

wrong number of arguments (given 1, expected 0)

Failure/Error: @configuration = Core::FeatureFlags::Configuration.new(configuration)

ArgumentError:
  wrong number of arguments (given 1, expected 0)
./lib/datadog/open_feature/native_evaluator.rb:14:in \`initialize&#39;
./lib/datadog/open_feature/native_evaluator.rb:14:in \`new&#39;
./lib/datadog/open_feature/native_evaluator.rb:14:in \`initialize&#39;
./spec/datadog/open_feature/native_evaluator_spec.rb:94:in \`new&#39;
...

New test introduced in this PR is flaky.

View in Flaky Test Management

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 067c53d | Docs | Datadog PR Page | Give us feedback!}

leoromanovsky added 8 commits July 29, 2026 19:23

Add canonical FFE fixture tests

3c5cd96

Align native FFE fixture assertions

c972a3e

Keep native FFE fixture error assertions strict

676739b

Update FFE fixture submodule

c33e981

Initialize canonical FFE fixtures in CI

5dedefc

fix(openfeature): align canonical fixture reasons

a85fc21

Merge canonical FFE fixtures into regex conformance stack

2bfa74a

Bring the canonical evaluator fixtures onto current master before testing the PR #21 regex additions. Environment: Datadog workspace

test(openfeature): cover isolated malformed flags

bd7e3c8

dd-octo-sts Bot added the openfeature A new component that provider an ability to configure feature flags label Aug 11, 2026

ci: configure submodule update cooldown

a082f80

fix(openfeature): load native extension explicitly

067c53d

Navigation Menu

Uh oh!

test(openfeature): cover isolated malformed flags - #6179

test(openfeature): cover isolated malformed flags#6179
leoromanovsky wants to merge 10 commits into
masterfrom
agent/isolate-malformed-flags

leoromanovsky commented Aug 11, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Aug 11, 2026

Uh oh!

datadog-datadog-us1-prod Bot commented Aug 11, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

leoromanovsky commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Validation

Uh oh!

dd-octo-sts Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Aug 11, 2026

Typing analysis

Untyped methods

Uh oh!

datadog-datadog-us1-prod Bot commented Aug 11, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

leoromanovsky commented Aug 11, 2026 •

edited

Loading

dd-octo-sts Bot commented Aug 11, 2026 •

edited

Loading

datadog-datadog-us1-prod Bot commented Aug 11, 2026 •

edited by datadog-official Bot

Loading