test(ffe): align invalid config metrics

manifests/golang.yml

@@ -1162,7 +1162,7 @@ manifest:
   tests/ffe/test_flag_eval_evp.py::Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed: missing_feature (FFL-2784)
   tests/ffe/test_flag_eval_evp.py::Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed: missing_feature (FFL-2784)
   tests/ffe/test_flag_eval_metrics.py: v2.8.0
-  tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex: irrelevant (Go validates regex at config load time)
+  tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Invalid_Regex_Default: irrelevant (Go validates regex at config load time)
   tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored: irrelevant (FFL-1980)
   tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
   tests/integrations/crossed_integrations/test_kafka.py::Test_Kafka:

manifests/nodejs.yml

@@ -1733,11 +1733,9 @@ manifest:
         "*": incomplete_test_app
         express4: *ref_5_99_0
   tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: bug (FFL-2313)
+  tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Invalid_Regex_Default::test_ffe_eval_metric_invalid_regex_default: bug (FFL-2313)
+  tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Invalid_Variant_Default::test_ffe_eval_metric_invalid_variant_default: bug (FFL-2313)
   tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: bug (FFL-2313)
-  ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex
-  : bug (FFL-2313)
-  ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch
-  : bug (FFL-2313)
   tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-2313)
   tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional: bug (FFL-1730)
   tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py::TestAnthropicApmMessages: *ref_5_71_0

manifests/python.yml

@@ -1354,6 +1354,8 @@ manifest:
   tests/ffe/test_exposures.py: v4.2.0-dev
   tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446)
   tests/ffe/test_flag_eval_metrics.py: v4.7.0
+  tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Invalid_Regex_Default: v4.14.0
+  tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Invalid_Variant_Default: v4.14.0
   tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored: irrelevant (FFL-1980)
   tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py::TestAnthropicApmMessages: v3.16.0
   tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages: v3.16.0

tests/ffe/README.md

@@ -96,22 +96,25 @@ Check if the wheel is on S3 by verifying the "Upload wheels to S3" job passed in
 
 The system tests cover:
 
-### Resolution Reasons (5 tests)
+### Resolution Reasons (7 tests)
 | Reason | Test | Scenario |
 |--------|------|----------|
 | `static` | `Test_FFE_Eval_Metric_Basic` | No rules, no shards (catch-all) |
 | `targeting_match` | `Test_FFE_Eval_Reason_Targeting` | Targeting rules match context |
 | `split` | `Test_FFE_Eval_Reason_Split` | 50/50 shard-based rollout |
 | `default` | `Test_FFE_Eval_Reason_Default` | Rules don't match |
+| `default` | `Test_FFE_Eval_Metric_Invalid_Regex_Default` | Invalid regex in per-flag configuration |
+| `default` | `Test_FFE_Eval_Metric_Invalid_Variant_Default` | Variant value doesn't match the declared type |
 | `disabled` | `Test_FFE_Eval_Reason_Disabled` | Flag is disabled |
 
-### Error Codes (5 tests)
+Invalid per-flag configuration returns the caller default without `error.type`.
+A malformed full configuration payload remains a `parse_error`.
+
+### Error Codes (3 tests)
 | Error Code | Test | Trigger |
 |------------|------|---------|
 | `flag_not_found` | `Test_FFE_Eval_Config_Exists_Flag_Missing` | Config exists, flag missing |
 | `type_mismatch` | `Test_FFE_Eval_Metric_Type_Mismatch` | Request boolean from string flag |
-| `parse_error` | `Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex` | Invalid regex pattern in condition |
-| `parse_error` | `Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch` | Variant value doesn't match declared type |
 | `provider_not_ready` | `Test_FFE_Eval_No_Config_Loaded` | No config loaded |
 
 ### Other Tests
@@ -180,9 +183,9 @@ The tests use `interfaces.agent.get_metrics()` to retrieve metrics from the agen
 
 ## Common Issues
 
-### 1. Test returns `parse_error` instead of expected reason
-- Check UFC fixture format (especially `totalShards` placement)
-- Verify the fixture matches `flags-v1.json` format in dd-trace-py
+### 1. Invalid per-flag configuration returns an error instead of the caller default
+- Verify that the evaluator isolates invalid flags during configuration compilation
+- Verify that the result omits `error.type`
 
 ### 2. Test returns `static` instead of `split`
 - Need multiple variations with different shard ranges

tests/ffe/test_flag_eval_metrics.py

@@ -334,8 +334,8 @@ def make_split_fixture(flag_key: str):
 def make_invalid_regex_fixture(flag_key: str, invalid_regex: str = "[invalid"):
     """Create a UFC fixture with an invalid regex pattern in a MATCHES condition.
 
-    This tests the PARSE_ERROR scenario where the configuration contains
-    a syntactically invalid regex pattern that fails during evaluation.
+    This tests per-flag configuration isolation. An invalid flag returns the
+    caller default without error metadata.
     """
     return {
         "createdAt": "2024-04-17T19:40:53.716Z",
@@ -376,8 +376,8 @@ def make_invalid_regex_fixture(flag_key: str, invalid_regex: str = "[invalid"):
 def make_variant_type_mismatch_fixture(flag_key: str):
     """Create a UFC fixture where the variant value doesn't match the declared type.
 
-    This tests the PARSE_ERROR scenario where the configuration declares a flag type
-    (e.g., INTEGER) but the variant value is incompatible (e.g., a string).
+    This tests per-flag configuration isolation when the declared flag type and
+    variant value are incompatible.
     This is a configuration error, not a runtime type conversion error.
     """
     return {
@@ -736,21 +736,16 @@ def test_ffe_eval_metric_numeric_to_integer(self):
 
 @scenarios.feature_flagging_and_experimentation
 @features.feature_flags_eval_metrics
-class Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex:
-    """Test that an invalid regex pattern produces error.type=parse_error.
+class Test_FFE_Eval_Metric_Invalid_Regex_Default:
+    """Test that an invalid regex pattern produces caller-default metrics.
 
-    This configures a flag with a MATCHES condition containing an invalid regex pattern
-    (e.g., "[invalid" which has an unclosed bracket). When the condition is evaluated,
-    the regex compilation fails and produces a parse_error.
-
-    Behavioral differences across SDKs:
-    - Python (libdatadog): Returns parse_error during evaluation
-    - Go: Validates regex at config load time, rejects config with invalid regex
+    Per-flag invalid configuration is isolated. The evaluation returns the caller
+    default without error metadata. A malformed full payload remains a parse error.
     """
 
-    def setup_ffe_eval_metric_parse_error_invalid_regex(self):
-        config_id = "ffe-eval-metric-parse-error"
-        self.flag_key = "eval-metric-parse-error-flag"
+    def setup_ffe_eval_metric_invalid_regex_default(self):
+        config_id = "ffe-eval-invalid-regex-default"
+        self.flag_key = "eval-invalid-regex-default-flag"
         rc.tracer_rc_state.reset().set_config(
             f"{RC_PATH}/{config_id}/config", make_invalid_regex_fixture(self.flag_key)
         ).apply()
@@ -767,8 +762,8 @@ def setup_ffe_eval_metric_parse_error_invalid_regex(self):
             },
         )
 
-    def test_ffe_eval_metric_parse_error_invalid_regex(self):
-        """Test that invalid regex produces error.type:parse_error."""
+    def test_ffe_eval_metric_invalid_regex_default(self):
+        """Test that invalid regex produces a default result without an error."""
         assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
 
         metrics = find_eval_metrics(self.flag_key)
@@ -777,31 +772,28 @@ def test_ffe_eval_metric_parse_error_invalid_regex(self):
         point = metrics[0]
         tags = point.get("tags", [])
 
-        assert get_tag_value(tags, "feature_flag.result.reason") == "error", (
-            f"Expected reason 'error' for parse error, got tags: {tags}"
-        )
-        assert get_tag_value(tags, "error.type") == "parse_error", (
-            f"Expected error.type 'parse_error', got tags: {tags}"
+        assert get_tag_value(tags, "feature_flag.result.reason") == "default", (
+            f"Expected reason 'default' for invalid regex, got tags: {tags}"
         )
+        assert get_tag_value(tags, "error.type") is None, f"Expected no error.type for invalid regex, got tags: {tags}"
 
 
 @scenarios.feature_flagging_and_experimentation
 @features.feature_flags_eval_metrics
-class Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch:
-    """Test that a variant value not matching declared flag type produces parse_error.
+class Test_FFE_Eval_Metric_Invalid_Variant_Default:
+    """Test that an invalid variant value produces caller-default metrics.
 
     This configures a flag as INTEGER type but gives the variant a string value.
-    When the configuration is validated during evaluation, this type mismatch
-    produces a parse_error (configuration is invalid).
+    Per-flag invalid configuration is isolated. The evaluation returns the caller
+    default without error metadata.
 
     This is different from Test_FFE_Eval_Metric_Type_Mismatch which tests
     runtime type conversion (e.g., evaluating a STRING flag as BOOLEAN).
-    This test validates that configuration errors are properly detected.
     """
 
-    def setup_ffe_eval_metric_parse_error_variant_type_mismatch(self):
-        config_id = "ffe-eval-variant-type-mismatch"
-        self.flag_key = "eval-variant-type-mismatch-flag"
+    def setup_ffe_eval_metric_invalid_variant_default(self):
+        config_id = "ffe-eval-invalid-variant-default"
+        self.flag_key = "eval-invalid-variant-default-flag"
         rc.tracer_rc_state.reset().set_config(
             f"{RC_PATH}/{config_id}/config", make_variant_type_mismatch_fixture(self.flag_key)
         ).apply()
@@ -818,8 +810,8 @@ def setup_ffe_eval_metric_parse_error_variant_type_mismatch(self):
             },
         )
 
-    def test_ffe_eval_metric_parse_error_variant_type_mismatch(self):
-        """Test that variant type mismatch produces error.type:parse_error."""
+    def test_ffe_eval_metric_invalid_variant_default(self):
+        """Test that an invalid variant produces a default result without an error."""
         assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
 
         metrics = find_eval_metrics(self.flag_key)
@@ -828,11 +820,11 @@ def test_ffe_eval_metric_parse_error_variant_type_mismatch(self):
         point = metrics[0]
         tags = point.get("tags", [])
 
-        assert get_tag_value(tags, "feature_flag.result.reason") == "error", (
-            f"Expected reason 'error' for variant type mismatch, got tags: {tags}"
+        assert get_tag_value(tags, "feature_flag.result.reason") == "default", (
+            f"Expected reason 'default' for invalid variant, got tags: {tags}"
         )
-        assert get_tag_value(tags, "error.type") == "parse_error", (
-            f"Expected error.type 'parse_error' for variant type mismatch, got tags: {tags}"
+        assert get_tag_value(tags, "error.type") is None, (
+            f"Expected no error.type for invalid variant, got tags: {tags}"
         )