Skip to content

test: isolate malformed flags during UFC ingestion - #26

Merged
leoromanovsky merged 8 commits into
mainfrom
agent/variant-type-mismatch-fixtures
Aug 11, 2026
Merged

test: isolate malformed flags during UFC ingestion#26
leoromanovsky merged 8 commits into
mainfrom
agent/variant-type-mismatch-fixtures

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Motivation

This PR adds regression coverage for bugs in malformed-flag handling. A single invalid flag can currently poison an otherwise usable UFC, fall through to a catch-all allocation, or be misreported as a normal default or missing flag. These behaviors hide configuration errors and unnecessarily break valid neighboring flags.

The corrected contract isolates each malformed flag during ingestion, keeps the rest of the UFC usable, and reports the rejected key accurately as ERROR / PARSE_ERROR.

Changes

  • Add fixtures for malformed variant values and invalid GT, ONE_OF, and IS_NULL operands.
  • Cover invalid regexes, configured SemVer comparands, allocation shapes, missing shards, invalid shard bounds, null shard ranges, and unknown operators.
  • Verify that valid flags in the same UFC continue evaluating normally.
  • Require evaluations of rejected flag keys to return the caller default with reason ERROR and error code PARSE_ERROR.
  • Reserve ERROR / FLAG_NOT_FOUND for keys that were not present in the UFC.
  • Make downstream compatibility checks visibly fail when proposed fixtures expose an implementation gap, while keeping those checks non-required.

Decisions

  • Validate each flag independently during UFC ingestion.
  • Exclude malformed flags from the active evaluation map without rejecting the full UFC.
  • Retain only the rejected key and its error for the current configuration; do not retain malformed flag objects.
  • Rebuild and atomically replace the active and rejected maps on every refresh. Fixing or deleting a malformed flag therefore removes its stale rejection entry.
  • Keep compatibility failures advisory: a red downstream check communicates work remaining in that consumer but does not block this fixture PR from merging.
  • Update downstream implementations and submodule references after this contract merges.

Validation

  • python3 ci/validate-fixtures.py — 266 cases across 37 files
  • bash -n ci/run-downstream-conformance.sh
  • git diff --check

@leoromanovsky leoromanovsky changed the title test: add variant type mismatch fixtures test: isolate malformed flags during UFC ingestion Aug 11, 2026
@leoromanovsky
leoromanovsky marked this pull request as ready for review August 11, 2026 01:12
@leoromanovsky
leoromanovsky requested a review from a team as a code owner August 11, 2026 01:12
@leoromanovsky
leoromanovsky requested review from pavlokhrebto and vjfridge and removed request for a team August 11, 2026 01:12
Comment thread evaluation-cases/test-case-invalid-regex-isolation.json Outdated
@leoromanovsky
leoromanovsky merged commit ea8b5cc into main Aug 11, 2026
2 of 10 checks passed
@leoromanovsky
leoromanovsky deleted the agent/variant-type-mismatch-fixtures branch August 11, 2026 01:26
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Aug 13, 2026
## Motivation

Malformed per-flag UFC configuration is already isolated during ingestion: libdatadog stores a rejection error for that key while compiling valid neighboring flags normally. The FFI translation incorrectly exposed that rejection as a normal DEFAULT result with no error code, hiding a configuration bug from callers.

This implements the contract merged in [DataDog/ffe-system-test-data#26](DataDog/ffe-system-test-data#26): evaluating a rejected key returns the caller default with ERROR / PARSE_ERROR, while valid neighboring flags remain usable.

## Changes

- Map FlagConfigurationInvalid to the FFI ERROR reason and PARSE_ERROR error code.
- Add regression coverage proving an invalid regex flag does not poison a valid neighboring flag.
- Bump the canonical fixture submodule to the revision merged in [DataDog/ffe-system-test-data#26](DataDog/ffe-system-test-data#26).
- Update the canonical adapter to expect PARSE_ERROR for rejected keys.
- Compare semantic versions by precedence so build metadata does not change equality or ordering.
- Correct the regex-ingestion documentation to match the parser behavior.

## Decisions

- Keep validation in libdatadog so the same Rust regex and UFC parsers used for evaluation own rejection decisions.
- Retain only the per-key rejection error in the compiled configuration, not the malformed flag object.
- Preserve FLAG_NOT_FOUND for keys absent from the configuration.
- Use SemVer precedence comparisons, which ignore build metadata as required by the specification.

## Validation

- [x] cargo test -p libdd-ffe
- [x] cargo test -p libdd-ffe-ffi rejected_flag_is_exposed_as_parse_error
- [x] cargo test -p libdd-ffe-test-suite --test canonical_fixtures
- [x] git diff --check

Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants