ci: exercise downstream regex conformance

[leoromanovsky]

Important

Evidence only. This stacked draft PR is not intended for merge. Red jobs are diagnostic evidence, not merge gates for PR #21. For v2, only disagreement on an accepted case is an authoring-contract failure; native acceptance of rejected syntax is informational.

Motivation

Re-run the targeting-regex fixture from PR #21 against real, current downstream evaluator code and test whether the fixture is actually universal.

The earlier stack run was not sufficient evidence: its runner captured non-zero unit-test exits but always returned success, and the workflow explicitly declined to enforce Java, JavaScript, and .NET. Those three consumers were already failing behind green checks.

This run uses the exact PR #21 head 8e8a6c26703fbd801c8877afdd80e078c5751b30. The fixture self-validator passes, its checked-in SHA-256 is 4f8ef95faec9faa3c2b0756ac61afe862e35c0646cba5375532a85335f209e8f, and it contains 75 unique v2 cases: 30 accepted and 45 rejected.

Changes

Each remote evidence branch was merged with the listed default-branch tip, pinned to PR #21 head 8e8a6c2, and changed to load and validate the new regex fixture before running the repository's production regex/evaluator path.

Consumer	Remote evidence branch tip	Default tip merged	Path exercised
Go	ee135c0	main@29425b7	Production loadRegex + matchesRegex, normalized pattern, Go expectations
Java	4baecdd0	master@35c4350	Production DDEvaluator.matchesRegex/normalization + java.util.regex.Pattern; accepted cases are strict and rejected cases are exercised without requiring native rejection
JavaScript	329bbb2	master@fe6a4a0	Full FlaggingProvider evaluation; accepted cases are strict and rejected cases are exercised without requiring native rejection
Python	9de22ea	main@3a825cc	Native config processing + full DataDogProvider evaluation, raw pattern, Rust rules expectations
Ruby	ab2e8e0	master@4850645	Full NativeEvaluator/libdatadog evaluation, raw pattern, Rust rules expectations
.NET	97d2fd3	master@b58e8e8	Full FeatureFlagsEvaluator; accepted cases are strict and rejected cases are exercised without requiring native rejection
PHP	14f15d28	master@3ef1113	Full native FFE bridge/libdatadog evaluation, raw pattern, Rust rules expectations
libdatadog	2c25337	main@63a1381	Shared production libdd_common::regex_engine::Regex, raw pattern, Rust rules expectations

All eight listed branch-tip commits are GitHub-verified. The stack runner also checks each remote branch's gitlink against the PR #21 SHA before executing the unit test, so a stale or merely aspirational branch cannot produce a report.

Exact CI commands

Consumer	Command
Go	go test ./openfeature -run '^TestTargetingRegexConformance$' -count=1
Java	./gradlew :products:feature-flagging:feature-flagging-api:cleanTest :products:feature-flagging:feature-flagging-api:test --tests datadog.trace.api.openfeature.DDEvaluatorTest.testRegexConformance --no-daemon
JavaScript	npm run test:openfeature -- --grep 'targeting regex conformance'
Python	scripts/run-tests --venv 14fc413 -- -- -vv -k targeting_regex_conformance
Ruby	bundle exec rspec spec/datadog/open_feature/native_evaluator_spec.rb -e 'targeting regex conformance'
.NET	dotnet test tracer/test/Datadog.Trace.Tests/Datadog.Trace.Tests.csproj --framework net10.0 --no-restore --filter 'FullyQualifiedName~Datadog.Trace.Tests.FeatureFlags.FeatureFlagsEvaluatorTests.RegexConformance' --verbosity minimal
PHP	make test_c TESTS=tests/ext/ffe/regex_conformance.phpt MAX_TEST_PARALLELISM=1
libdatadog	cargo +1.87.0 test -p libdd-ffe-test-suite --test targeting_regex_conformance evaluates_targeting_regex_conformance_fixture -- --exact --nocapture

CI evidence

Current v2 workflow run 31738632407 uses PR0 commit 8e8a6c2 and the eight updated branch tips above. Attempt 2 completed successfully: all eight downstream jobs passed.

Previous v1 evidence

Workflow run 31709709467, attempt 2 checked out the earlier evidence-branch tips, resolved the stacked PR base to ab0a81f, verified those branch gitlinks, and preserved the actual unit-test exit code.

Consumer	Unit-test output	Result
Go	Fixture shape asserts 75 unique cases; focused test exit 0	Pass
Java	66 unique cases; 7 unique mismatches. Gradle retry reports 264 tests completed, 28 failed because each mismatch is attempted four times.	Fail
JavaScript	53 passing, 9 pending, 13 failing	Fail
Python	75 passed, 706 deselected, 1 warning	Pass
Ruby	75 examples, 0 failures	Pass
.NET	Failed: 13, Passed: 54, Total: 67; totals are 66 common conformance cases plus the passing 75-case fixture-shape test	Fail
PHP	One PHPT passes while sweeping all 75 cases; all 19 expected compile failures are observed as flag-load or evaluation rejection with 0 mismatches	Pass
libdatadog	1 passed; 0 failed; the test loops over and validates all 75 cases	Pass

Java's 7 unique mismatches are Python-style named groups plus positive/negative lookahead, positive/negative lookbehind, numeric backreferences, and atomic groups.

JavaScript's 13 mismatches are the two named-group forms, POSIX alpha, Unicode letter, two leading-closing-bracket cases, byte/boundary escapes, four lookaround cases, and numeric backreferences. Nine engine-divergent cases are explicitly reported as pending instead of disappearing from the test count.

.NET's 13 mismatches are the two named-group forms, POSIX alpha, three inline-Unicode forms, boundary escape in a character class, four lookaround cases, numeric backreferences, and atomic groups.

Decisions

The v1 evidence showed that nine accepted cases across five syntax families were not evaluated consistently by every shipped SDK. Contract v2 reclassifies those cases as rejected: named capture groups, POSIX classes, Unicode property classes, combined or scoped Unicode flags, and leading-literal-] character classes.

• Every shipped SDK must agree on accepted cases.
• Native acceptance of rejected syntax is not an SDK production bug when dd-source and web-ui block that syntax from new authoring.
• Existing stored uses still require a release audit and customer remediation before backend enforcement.
• No tracer branch in this stack exercises RE2JS. dd-trace-js exercises its real server provider/native ECMAScript behavior.
• Python, Ruby, PHP, and libdatadog are correlated evidence for the shared Rust implementation, not four independent regex engines. Go is the independent modeled-engine confirmation.
• The Java, JavaScript, and .NET adapters now enforce accepted cases while exercising rejected cases without requiring every native engine to reject them. Go and the Rust-backed consumers keep exact per-engine assertions.
• The standalone regex fixture tests syntax, compile, and match behavior. Invalid-regex flag isolation remains the responsibility of evaluation-cases/test-case-invalid-regex-isolation.json and the ordinary downstream compatibility suites.

The stack remains evidence-only and should stay draft.

[datadog-datadog-prod-us1]

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: aa57122 | Docs | Datadog PR Page | Give us feedback!}