test(openfeature): add canonical FFE fixture tests - #8441
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #8441 +/- ##
========================================
Coverage 98.55% 98.55%
========================================
Files 966 966
Lines 139409 139583 +174
Branches 12389 12100 -289
========================================
+ Hits 137396 137570 +174
Misses 2013 2013 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Overall package sizeSelf size: 7.99 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🔄 Datadog auto-retried 1 job - 1 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: 780f5ad | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-08-10 20:51:16 Comparing candidate commit 780f5ad in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2322 metrics, 36 unstable metrics.
|
f3d8f67 to
43713f0
Compare
## Motivation Use the shared FFE fixture corpus. This prevents copied Go fixtures from drifting from other SDKs. The same migration is merged in [Java](DataDog/dd-trace-java#11355) and [libdatadog](DataDog/libdatadog#1979). Related migrations are open for [Python](DataDog/dd-trace-py#19390), [JavaScript](DataDog/dd-trace-js#8441), [Ruby](DataDog/dd-trace-rb#5742), and [.NET](DataDog/dd-trace-dotnet#8616). ## Changes and Decisions - Add `DataDog/ffe-system-test-data` as an OpenFeature test submodule. - Read all canonical configuration and evaluation cases from the submodule. - Parse and validate flags independently so one invalid flag does not reject valid neighbors. - Assert canonical values and reasons, including temporal defaults and missing-flag errors. - Initialize submodules in Go test workflows and configure weekly Dependabot updates. Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
…canonical-fixtures-20260512
leoromanovsky
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 80676d91ef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| function loadFixtureCases () { | ||
| const fixtureFiles = fs.readdirSync(fixtureCaseDir).filter(file => file.endsWith('.json')).sort() |
There was a problem hiding this comment.
After a normal non-recursive clone—or when using a GitHub source archive—the new submodule directory has no evaluation-cases child, so this readdirSync throws ENOENT while the spec is loading and npm run test:openfeature cannot run at all. Only the dedicated CI checkout initializes the submodule, and no repository setup command or documentation initializes it for developers; add the initialization to the normal setup path or provide an actionable fixture bootstrap before loading the suite.
Useful? React with 👍 / 👎.
| sinon.assert.notCalled(mockChannel.publish) | ||
| }) | ||
|
|
||
| it('preserves TYPE_MISMATCH when the requested type differs from the flag type', async () => { |
There was a problem hiding this comment.
why are these needed if they were in the fixtures?
| }) | ||
| } | ||
|
|
||
| function integerFlagConfiguration (value) { |
## Motivation Use the shared FFE fixture corpus so the .NET evaluator is checked against the same behavior as the other tracer implementations. This reduces fixture drift and gives us a repeatable way to expose and correct evaluator bugs when new canonical cases are added. The same fixture corpus is used by [Java](DataDog/dd-trace-java#11355), [libdatadog](DataDog/libdatadog#1979), [Go](DataDog/dd-trace-go#4753), [Python](DataDog/dd-trace-py#19390), [JavaScript](DataDog/dd-trace-js#8441), and [Ruby](DataDog/dd-trace-rb#5742). ## Changes - Replace the legacy copied fixtures with a generated, checked-in snapshot from `DataDog/ffe-system-test-data`. - Record the exact upstream commit in `SOURCE.md`. - Add a script that fetches, validates, and copies the canonical configuration and evaluation cases. - Add a weekly and manually dispatchable workflow that opens a signed draft dependency PR only when fixture contents have changed. - Parse flags independently so malformed flags do not reject valid neighbors. - Return `FLAG_NOT_FOUND` for missing flags and classify temporal, static, and split allocations. - Assert canonical values and reasons through the existing .NET unit-test suite. ## Fixture update flow When we add or change shared evaluator behavior, I imagine the flow working like this: 1. Add the new configuration and evaluation cases to [`DataDog/ffe-system-test-data`](https://github.com/DataDog/ffe-system-test-data) and review the expected behavior there. 2. The weekly updater, or a manually dispatched run for a specific ref, fetches the canonical repository and compares its fixture contents with the checked-in .NET snapshot. 3. If nothing changed, the workflow exits without opening or updating a PR. 4. If fixtures changed, the workflow copies them into this repository, records the source commit, and opens a signed draft PR with the normal dependency labels. 5. The .NET unit tests run against the updated cases. New tests may intentionally fail when they catch an evaluator bug or unsupported behavior. 6. Address those evaluator failures in the same dependency PR, keeping the fixture expectations unchanged unless the shared expectation itself is incorrect. 7. Merge the update once the .NET evaluator satisfies the new canonical cases. This keeps new behavior explicit and reviewable: fixture changes land in the canonical repository first, and each tracer then gets a visible compatibility PR rather than silently changing at build time. ## Decisions - `DataDog/ffe-system-test-data` remains the canonical source of shared evaluator behavior. - Keep the generated snapshot checked in so local and CI unit tests do not require network access or submodule initialization. - Use a scheduled dependency-update workflow instead of a git submodule. - Treat failures introduced by new canonical fixtures as useful regression signals and fix the evaluator as part of accepting the update. - Do not create a PR when the canonical fixture contents are unchanged, even if the upstream repository has unrelated commits.
1 participant
Motivation
Use the shared FFE fixture corpus. This prevents copied JavaScript fixtures from drifting from other SDKs.
Identical submodule-backed canonical-fixture implementations are already merged in
dd-trace-javaanddd-trace-go. The shared evaluator implementation is also merged inlibdatadog.Related migrations are open for Python, Ruby, and .NET.
Changes
DataDog/ffe-system-test-dataas a submodule under the OpenFeature tests.Decisions
ERRORfor missing flags, and classify temporal, static, and split allocations.