Add canonical FFE fixture tests by leoromanovsky · Pull Request #5742 · DataDog/dd-trace-rb

leoromanovsky · 2026-05-12T16:41:48Z

Motivation

Use the shared FFE fixture corpus. This prevents copied Ruby fixtures from drifting from other SDKs.

Identical submodule-backed canonical-fixture implementations are already merged in dd-trace-java and dd-trace-go. The shared evaluator implementation is also merged in libdatadog.

Related migrations are open for Python, JavaScript, and .NET.

Changes

Add DataDog/ffe-system-test-data as a submodule under the OpenFeature specs.
Load the canonical configuration and evaluation cases in the native evaluator spec.
Assert canonical values and reasons, and assert optional metadata only when fixtures define it.
Initialize the submodule in the unit-test workflow.
Exclude .gitmodules from the packaged gem file set and configure weekly Dependabot updates.

Decisions

Reuse the shared fixture repository instead of maintaining tracer-specific copies.
Keep libdatadog invalid-flag results as caller defaults instead of parse errors.

dd-octo-sts · 2026-05-12T16:42:02Z

👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description.

If changes need to be present in CHANGELOG.md you can state it this way

**Change log entry**

Yes. A brief summary to be placed into the CHANGELOG.md

(possible answers Yes/Yep/Yeah)

Or you can opt out like that

**Change log entry**

None.

(possible answers No/Nope/None)

^{Visited at: 2026-08-10 17:53:55 UTC}

datadog-prod-us1-5 · 2026-05-12T18:45:16Z

⚠️ Warnings

❄️ 236 New flaky tests detected

Datadog::OpenFeature::NativeEvaluator canonical FFE fixtures evaluates test-case-boolean-false-assignment.json[0] from rspec

Datadog::OpenFeature::NativeEvaluator canonical FFE fixtures evaluates test-case-boolean-false-assignment.json[1] from rspec

View in Flaky Test Management

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 88.75%
• Overall Coverage: 89.97% (+0.00%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f8eabf7 | Docs | Datadog PR Page | Give us feedback!}

## Motivation Use the shared FFE fixture corpus. This prevents copied Go fixtures from drifting from other SDKs. The same migration is merged in [Java](DataDog/dd-trace-java#11355) and [libdatadog](DataDog/libdatadog#1979). Related migrations are open for [Python](DataDog/dd-trace-py#19390), [JavaScript](DataDog/dd-trace-js#8441), [Ruby](DataDog/dd-trace-rb#5742), and [.NET](DataDog/dd-trace-dotnet#8616). ## Changes and Decisions - Add `DataDog/ffe-system-test-data` as an OpenFeature test submodule. - Read all canonical configuration and evaluation cases from the submodule. - Parse and validate flags independently so one invalid flag does not reject valid neighbors. - Assert canonical values and reasons, including temporal defaults and missing-flag errors. - Initialize submodules in Go test workflows and configure weekly Dependabot updates. Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>

…canonical-fixtures-20260512

dd-octo-sts · 2026-08-10T18:54:08Z

Typing analysis

Note: Ignored files are excluded from the next sections.

Untyped methods

This PR clears 2 partially typed methods. It increases the percentage of typed methods from 69.52% to 69.64% (+0.12%).

Partially typed methods (+0-2)

✅ Cleared:

sig/datadog/open_feature/native_evaluator.rbs:10
└── def get_assignment: (
        ::String flag_key,
        default_value: untyped,
        context: ::OpenFeature::SDK::EvaluationContext::fields_t,
        expected_type: ::Symbol
      ) -> (Core::FeatureFlags::ResolutionDetails | ResolutionDetails)
sig/datadog/open_feature/native_evaluator.rbs:21
└── def invalid_flag_configuration_error: (untyped default_value) -> ResolutionDetails

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f8eabf79ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-10T19:59:32Z

+        return false unless ["MATCHES", "NOT_MATCHES"].include?(condition["operator"])
+        return false unless condition["value"].is_a?(String)
+
+        Regexp.new(condition["value"])


Validate regexes with the evaluator's regex dialect

When a condition uses syntax on which Ruby's Regexp and libdatadog's regex engine disagree, this check produces the wrong resolution: for example, Rust-regex flag syntax such as scoped Unicode-mode options can be valid to libdatadog but rejected here, causing a valid flag to return PARSE_ERROR without evaluation; conversely, lookaround such as (?=foo) is accepted here but unsupported by the evaluator, so the invalid configuration falls through as a non-error DEFAULT now that the previous invalid-configuration translation was removed. Validate with the same regex parser as libdatadog, or preserve translation of its invalid-configuration result.

Useful? React with 👍 / 👎.

The general gist of this AI review comment is: Ruby's regular expressions might not be 100% compatible with regular expressions created for other environments.
Ruby uses a slightly modified version of the Onigmo's regex library: https://github.com/ruby/ruby/blob/master/regsyntax.c
It's a superset of https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap09.html (POSIX Extended Regular Expressions (ERE)).

This is likely outside the scope of this PR, but relevant to feature flags as a whole.

The general gist of this AI review comment is: Ruby's regular expressions might not be 100% compatible with regular expressions created for other environments.

Totally understand - all languages have different capabilities they support. We are working on a set of conformance tests that give us a universal amount of support (DataDog/ffe-system-test-data#21) so we'll be editing this in a follow up PR as the specification lands to constraint them.

chatgpt-codex-connector · 2026-08-10T19:59:32Z

+            error_message: "Variant value does not match the declared variation type"
+          )
+        end
+        if @invalid_regex_flags.include?(flag_key)


Preserve type mismatch precedence for invalid-regex flags

When an invalid-regex flag is requested through an API for the wrong type—for example, a declared STRING flag fetched with fetch_boolean_value—this unconditional lookup returns PARSE_ERROR before the native evaluator can return TYPE_MISMATCH. The variant-value validation immediately above avoids this by associating each invalid flag with its declared expected types; the regex validation needs the same type guard so malformed regexes do not change wrong-type resolution semantics.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-10T19:59:32Z

+  fixture_root = File.expand_path("../open_feature/ffe-system-test-data", __dir__)
+  fixture_files = Dir[File.join(fixture_root, "evaluation-cases", "*.json")].sort
+
+  raise "FFE fixture submodule is missing or empty" if fixture_files.empty?


Document initialization of the required fixture submodule

In a normal fresh clone without --recurse-submodules, this directory is empty, so loading spec/datadog/core/feature_flags_spec.rb raises before any examples run and the documented local test:core_with_libdatadog_api workflow cannot be used after the repository's normal setup steps. CI checkouts were updated to fetch the submodule, but local/container setup was not; add git submodule update --init to the documented setup path or avoid making the suite fail solely because the fixture checkout is absent.

AGENTS.md reference: AGENTS.md:L7-L15

Useful? React with 👍 / 👎.

marcotc

I wasn't able to finish my review today, I'll continue tomorrow.

marcotc · 2026-08-10T21:43:40Z

-          result.error_code.nil? &&
-          result.error_message == INVALID_FLAG_CONFIGURATION_ERROR_MESSAGE
+      def find_flag_validation_errors(configuration)
+        flags = JSON.parse(configuration)["flags"]


Is the JSON guaranteed to:

Be valid?

Contain a JSON object as the root?

We have a rescue for invalid JSON parsing, 1, which tells me we are confident receiving invalid JSONs is likely, aka we don't trust the JSON received here.
There's no error handling in case the scenario 2 above is not true.

We should not add error handling unnecessarily, since it adds overhead and code maintenance; but we should add necessary error handling if needed.

So my question here is: is it needed?

marcotc · 2026-08-10T21:45:21Z

      # @return [Core::FeatureFlags::ResolutionDetails] The assignment for the flag
      def get_assignment(flag_key, default_value:, expected_type:, context:)
-        result = @configuration.get_assignment(flag_key, expected_type, context)
+        if @variant_type_mismatch_flags.fetch(flag_key, []).include?(expected_type)


We are creating a new object ([]) on every call, to make include? work nicely, but we don't need that extra short-lived, brand new object here.

Yea this looks like an undesirable design, thanks for mentioning. I will shift this implementation down to libdatadog; please hold off on further review I'll re-ping when a new libdatadog version is ready.

marcotc · 2026-08-10T21:49:03Z

+      end
+
+      def expected_types_for(variation_type)
+        {


This method is creating this lookup hash for every method invocation, when it can be a static constant instead.

marcotc · 2026-08-10T21:54:14Z

+            end
+          end
+
+          invalid_regexes[flag_key] = true if flag_has_invalid_regex?(flag)


invalid_regexes looks like a set instead of a hash, since the lookups are all for the simple presence of elements.

marcotc · 2026-08-10T23:55:39Z

+        return false unless ["MATCHES", "NOT_MATCHES"].include?(condition["operator"])
+        return false unless condition["value"].is_a?(String)
+
+        Regexp.new(condition["value"])


The general gist of this AI review comment is: Ruby's regular expressions might not be 100% compatible with regular expressions created for other environments.
Ruby uses a slightly modified version of the Onigmo's regex library: https://github.com/ruby/ruby/blob/master/regsyntax.c
It's a superset of https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap09.html (POSIX Extended Regular Expressions (ERE)).

This is likely outside the scope of this PR, but relevant to feature flags as a whole.

## Motivation Use the shared FFE fixture corpus so the .NET evaluator is checked against the same behavior as the other tracer implementations. This reduces fixture drift and gives us a repeatable way to expose and correct evaluator bugs when new canonical cases are added. The same fixture corpus is used by [Java](DataDog/dd-trace-java#11355), [libdatadog](DataDog/libdatadog#1979), [Go](DataDog/dd-trace-go#4753), [Python](DataDog/dd-trace-py#19390), [JavaScript](DataDog/dd-trace-js#8441), and [Ruby](DataDog/dd-trace-rb#5742). ## Changes - Replace the legacy copied fixtures with a generated, checked-in snapshot from `DataDog/ffe-system-test-data`. - Record the exact upstream commit in `SOURCE.md`. - Add a script that fetches, validates, and copies the canonical configuration and evaluation cases. - Add a weekly and manually dispatchable workflow that opens a signed draft dependency PR only when fixture contents have changed. - Parse flags independently so malformed flags do not reject valid neighbors. - Return `FLAG_NOT_FOUND` for missing flags and classify temporal, static, and split allocations. - Assert canonical values and reasons through the existing .NET unit-test suite. ## Fixture update flow When we add or change shared evaluator behavior, I imagine the flow working like this: 1. Add the new configuration and evaluation cases to [`DataDog/ffe-system-test-data`](https://github.com/DataDog/ffe-system-test-data) and review the expected behavior there. 2. The weekly updater, or a manually dispatched run for a specific ref, fetches the canonical repository and compares its fixture contents with the checked-in .NET snapshot. 3. If nothing changed, the workflow exits without opening or updating a PR. 4. If fixtures changed, the workflow copies them into this repository, records the source commit, and opens a signed draft PR with the normal dependency labels. 5. The .NET unit tests run against the updated cases. New tests may intentionally fail when they catch an evaluator bug or unsupported behavior. 6. Address those evaluator failures in the same dependency PR, keeping the fixture expectations unchanged unless the shared expectation itself is incorrect. 7. Merge the update once the .NET evaluator satisfies the new canonical cases. This keeps new behavior explicit and reviewable: fixture changes land in the canonical repository first, and each tracer then gets a visible compatibility PR rather than silently changing at build time. ## Decisions - `DataDog/ffe-system-test-data` remains the canonical source of shared evaluator behavior. - Keep the generated snapshot checked in so local and CI unit tests do not require network access or submodule initialization. - Use a scheduled dependency-update workflow instead of a git submodule. - Treat failures introduced by new canonical fixtures as useful regression signals and fix the evaluator as part of accepting the update. - Do not create a PR when the canonical fixture contents are unchanged, even if the upstream repository has unrelated commits.

leoromanovsky added 4 commits July 29, 2026 19:23

Add canonical FFE fixture tests

3c5cd96

Align native FFE fixture assertions

c972a3e

Keep native FFE fixture error assertions strict

676739b

Update FFE fixture submodule

c33e981

leoromanovsky force-pushed the leo.romanovsky/ffe-canonical-fixtures-20260512 branch from 5d09d29 to c33e981 Compare July 30, 2026 01:34

leoromanovsky added 2 commits July 29, 2026 19:38

Initialize canonical FFE fixtures in CI

5dedefc

fix(openfeature): align canonical fixture reasons

a85fc21

dd-octo-sts Bot added the openfeature A new component that provider an ability to configure feature flags label Jul 30, 2026

leoromanovsky added 4 commits August 10, 2026 17:23

Merge remote-tracking branch 'origin/master' into leo.romanovsky/ffe-…

0f12f86

…canonical-fixtures-20260512

test(openfeature): load native extension for fixtures

d0dfcb5

fix(openfeature): report invalid variant values as parse errors

2c50aba

test(openfeature): fix FFE fixture CI coverage

6b60401

leoromanovsky added 2 commits August 10, 2026 19:01

ci: initialize FFE fixtures in native test jobs

a2eae46

fix(openfeature): report invalid regex as parse error

f8eabf7

leoromanovsky marked this pull request as ready for review August 10, 2026 19:49

leoromanovsky requested review from a team as code owners August 10, 2026 19:49

leoromanovsky requested review from sameerank and vjfridge August 10, 2026 19:49

chatgpt-codex-connector Bot reviewed Aug 10, 2026

View reviewed changes

marcotc reviewed Aug 10, 2026

View reviewed changes

leoromanovsky marked this pull request as draft August 11, 2026 01:26

Uh oh!

Conversation

leoromanovsky commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Uh oh!

dd-octo-sts Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-5 Bot commented May 12, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

dd-octo-sts Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Typing analysis

Untyped methods

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

marcotc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

leoromanovsky commented May 12, 2026 •

edited

Loading

dd-octo-sts Bot commented May 12, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented May 12, 2026 •

edited by datadog-official Bot

Loading

dd-octo-sts Bot commented Aug 10, 2026 •

edited

Loading