Skip to content

test: define malformed condition parse errors #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
leoromanovsky wants to merge 1 commit into from
Closed

test: define malformed condition parse errors #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,11 @@ Thank you for your interest in contributing to the FFE system test data reposito
"defaultValue": "<default value>",
"targetingKey": "<user identifier>",
"attributes": { "<key>": "<value>" },
"result": { "value": "<expected value>", "reason": "STATIC|SPLIT|TARGETING_MATCH|DEFAULT|ERROR|DISABLED" }
"result": {
"value": "<expected value>",
"reason": "STATIC|SPLIT|TARGETING_MATCH|DEFAULT|ERROR|DISABLED",
"errorCode": "<optional OpenFeature error code>"
}
}
]
```
Expand Down
4 changes: 3 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ Each evaluation case uses a universal schema with the following fields:
| `attributes` | object | Additional context attributes for targeting rules |
| `result.value` | any | The expected evaluation result value |
| `result.reason` | string | The expected OpenFeature reason: `STATIC`, `SPLIT`, `TARGETING_MATCH`, `DEFAULT`, `ERROR`, `DISABLED` |
| `result.errorCode` | string | Optional OpenFeature error code, such as `PARSE_ERROR` or `FLAG_NOT_FOUND` |

Example:

Expand Down Expand Up @@ -135,8 +136,9 @@ python3 ci/validate-fixtures.py
| `test-case-flag-with-empty-string.json` | Flag with empty string in configuration |
| `test-case-integer-flag.json` | Integer-typed flag evaluation |
| `test-case-kill-switch-flag.json` | Kill switch (emergency off) flag |
| `test-case-invalid-condition-operands.json` | Invalid configured operands for GT, ONE_OF, and IS_NULL produce parse errors without poisoning valid flags |
| `test-case-invalid-shard-bounds-isolation.json` | Flags with shard bounds outside Rust/schema integer ranges are ignored without poisoning valid flags |
| `test-case-invalid-regex-isolation.json` | Flag with an invalid regular expression is ignored without poisoning valid flags |
| `test-case-invalid-regex-isolation.json` | Flag with an invalid regular expression produces a parse error without poisoning the configuration |
| `test-case-malformed-flag-isolation.json` | Structurally malformed flag is ignored without poisoning valid flags |
| `test-case-microsecond-date-flag.json` | Flag with microsecond-precision date targeting |
| `test-case-missing-split-shards-isolation.json` | Flag with a split missing required `shards` is ignored without poisoning valid flags |
Expand Down
59 changes: 59 additions & 0 deletions evaluation-cases/test-case-invalid-condition-operands.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
[
{
"description": "GT requires a numeric configured operand. A string operand must produce a parse error rather than falling through to the catch-all allocation.",
"attributes": {
"age": 42
},
"defaultValue": "default",
"flag": "invalid-gt-condition-flag",
"result": {
"errorCode": "PARSE_ERROR",
"reason": "ERROR",
"value": "default"
},
"targetingKey": "invalid-gt",
"variationType": "STRING"
},
{
"description": "ONE_OF requires an array configured operand. A scalar operand must produce a parse error rather than falling through to the catch-all allocation.",
"attributes": {
"role": "admin"
},
"defaultValue": "default",
"flag": "invalid-one-of-condition-flag",
"result": {
"errorCode": "PARSE_ERROR",
"reason": "ERROR",
"value": "default"
},
"targetingKey": "invalid-one-of",
"variationType": "STRING"
},
{
"description": "IS_NULL requires a boolean configured operand. A string operand must produce a parse error rather than falling through to the catch-all allocation.",
"attributes": {
"deleted_at": null
},
"defaultValue": "default",
"flag": "invalid-is-null-condition-flag",
"result": {
"errorCode": "PARSE_ERROR",
"reason": "ERROR",
"value": "default"
},
"targetingKey": "invalid-is-null",
"variationType": "STRING"
},
{
"description": "A valid flag in the same configuration must still evaluate after malformed condition operands are isolated.",
"attributes": {},
"defaultValue": "default",
"flag": "valid-flag-after-invalid-config",
"result": {
"reason": "STATIC",
"value": "expected"
},
"targetingKey": "valid-neighbor",
"variationType": "STRING"
}
]
5 changes: 3 additions & 2 deletions evaluation-cases/test-case-invalid-regex-isolation.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
[
{
"description": "The malformed flag uses the invalid regex *@datadoghq.com before a catch-all allocation. SDKs must ignore the whole flag and return the caller default.",
"description": "The malformed flag uses the invalid regex *@datadoghq.com before a catch-all allocation. SDKs must return a parse error without poisoning the full configuration.",
"attributes": {
"email": "harish@datadoghq.com"
},
"defaultValue": "default",
"flag": "invalid-regex-flag",
"result": {
"reason": "DEFAULT",
"errorCode": "PARSE_ERROR",
"reason": "ERROR",
"value": "default"
},
"targetingKey": "harish",
Expand Down
1 change: 1 addition & 0 deletions evaluation-cases/test-case-semver-validation-flag.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@
"defaultValue": "unknown",
"flag": "semver-invalid-comparand-test",
"result": {
"errorCode": "PARSE_ERROR",
"reason": "ERROR",
"value": "unknown"
},
Expand Down
152 changes: 151 additions & 1 deletion ufc-config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3734,7 +3734,7 @@
"key": "invalid-regex-flag",
"enabled": true,
"variationType": "STRING",
"comment": "This flag deliberately uses the invalid regex *@datadoghq.com. SDKs must ignore this flag rather than falling through to a later allocation or rejecting the whole config.",
"comment": "This flag deliberately uses the invalid regex *@datadoghq.com. SDKs must report a parse error rather than falling through to a later allocation or rejecting the whole config.",
"variations": {
"invalid-regex-match": {
"key": "invalid-regex-match",
Expand Down Expand Up @@ -3780,6 +3780,156 @@
}
]
},
"invalid-gt-condition-flag": {
"key": "invalid-gt-condition-flag",
"enabled": true,
"variationType": "STRING",
"comment": "This flag deliberately gives GT a string operand. SDKs must report a parse error rather than falling through to the catch-all allocation.",
"variations": {
"trap": {
"key": "trap",
"value": "trap"
},
"catch-all": {
"key": "catch-all",
"value": "catch-all"
}
},
"allocations": [
{
"key": "invalid-gt-allocation",
"rules": [
{
"conditions": [
{
"attribute": "age",
"operator": "GT",
"value": "not-a-number"
}
]
}
],
"splits": [
{
"variationKey": "trap",
"shards": []
}
],
"doLog": true
},
{
"key": "invalid-gt-catch-all",
"rules": [],
"splits": [
{
"variationKey": "catch-all",
"shards": []
}
],
"doLog": true
}
]
},
"invalid-one-of-condition-flag": {
"key": "invalid-one-of-condition-flag",
"enabled": true,
"variationType": "STRING",
"comment": "This flag deliberately gives ONE_OF a scalar operand. SDKs must report a parse error rather than falling through to the catch-all allocation.",
"variations": {
"trap": {
"key": "trap",
"value": "trap"
},
"catch-all": {
"key": "catch-all",
"value": "catch-all"
}
},
"allocations": [
{
"key": "invalid-one-of-allocation",
"rules": [
{
"conditions": [
{
"attribute": "role",
"operator": "ONE_OF",
"value": "admin"
}
]
}
],
"splits": [
{
"variationKey": "trap",
"shards": []
}
],
"doLog": true
},
{
"key": "invalid-one-of-catch-all",
"rules": [],
"splits": [
{
"variationKey": "catch-all",
"shards": []
}
],
"doLog": true
}
]
},
"invalid-is-null-condition-flag": {
"key": "invalid-is-null-condition-flag",
"enabled": true,
"variationType": "STRING",
"comment": "This flag deliberately gives IS_NULL a non-boolean operand. SDKs must report a parse error rather than falling through to the catch-all allocation.",
"variations": {
"trap": {
"key": "trap",
"value": "trap"
},
"catch-all": {
"key": "catch-all",
"value": "catch-all"
}
},
"allocations": [
{
"key": "invalid-is-null-allocation",
"rules": [
{
"conditions": [
{
"attribute": "deleted_at",
"operator": "IS_NULL",
"value": "not-a-boolean"
}
]
}
],
"splits": [
{
"variationKey": "trap",
"shards": []
}
],
"doLog": true
},
{
"key": "invalid-is-null-catch-all",
"rules": [],
"splits": [
{
"variationKey": "catch-all",
"shards": []
}
],
"doLog": true
}
]
},
"malformed-allocations-flag": {
"key": "malformed-allocations-flag",
"enabled": true,
Expand Down
Loading