fix(ffe): report malformed flags as parse errors by leoromanovsky · Pull Request #2343 · DataDog/libdatadog

leoromanovsky · 2026-08-11T16:35:03Z

Motivation

Ruby, PHP, and Python share libdatadog's FFE evaluator. The updated canonical corpus requires malformed flag keys to report parse errors without blocking valid siblings, and exposes a SemVer precedence bug for build metadata.

Changes

bump ffe-system-test-data to ea8b5cc5ce335109f11f3efbc5fd608f98a3ca54
expose the existing per-flag invalid-configuration sentinel as ERROR/PARSE_ERROR through the FFI
keep unrecognized flags on ERROR/FLAG_NOT_FOUND
compare SemVer values by precedence so build metadata is ignored
add focused FFI and SemVer regressions and update the canonical harness

Decisions

retain FlagConfigurationInvalid as the per-flag sentinel; reserve ConfigurationParseError for whole-payload failures
keep independent flag compilation and immutable full-map replacement as the isolation boundary
put shared behavior here instead of duplicating evaluator logic in Ruby, PHP, and Python
keep the separate regex-fixture proposal in ffe-system-test-data#21 out of scope

Validation

cargo test -p libdd-ffe -p libdd-ffe-ffi -p libdd-ffe-test-suite --no-fail-fast
cargo +nightly-2026-07-26 fmt --all -- --check
cargo +stable clippy -p libdd-ffe -p libdd-ffe-ffi -p libdd-ffe-test-suite --all-targets -- -D warnings
cargo ffi-test built the release FFI artifacts and 14 example targets before an intentional stop during unrelated examples
git diff --check

github-actions · 2026-08-11T16:37:22Z

📚 Documentation Check Results

⚠️ 929 documentation warning(s) found

📦 `libdd-ffe-ffi` - 545 warning(s)

📦 `libdd-ffe-test-suite` - ✅ No warnings

📦 `libdd-ffe` - 384 warning(s)

Updated: 2026-08-11 16:37:21 UTC | Commit: 5bade8b | missing-docs job results

github-actions · 2026-08-11T16:39:03Z

🔒 Cargo Deny Results

⚠️ 11 issue(s) found, showing only errors (advisories, bans, sources)

📦 `libdd-ffe-ffi` - 3 error(s)

Show output

error[unmaintained]: Bincode is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:30:1
   │
30 │ bincode 1.3.3 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2025-0141
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2025-0141
   ├ Due to a doxxing and harassment incident, the bincode team has taken the decision to cease development permanently.
     
     The team considers version 1.3.3 a complete version of bincode that is not in need of any updates.
     
     ## Alternatives to consider
     
     * [wincode](https://crates.io/crates/wincode)
     * [postcard](https://crates.io/crates/postcard)
     * [bitcode](https://crates.io/crates/bitcode)
     * [rkyv](https://crates.io/crates/rkyv)
   ├ Announcement: https://git.sr.ht/~stygianentity/bincode/tree/v3.0/item/README.md
   ├ Solution: No safe upgrade is available!
   ├ bincode v1.3.3
     └── (dev) libdd-ffe v1.0.0
         └── libdd-ffe-ffi v1.0.2

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:194:1
    │
194 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── (dev) libdd-common v5.1.1
      │   ├── libdd-capabilities-impl v4.0.0
      │   │   ├── (dev) libdd-ffe v1.0.0
      │   │   │   └── libdd-ffe-ffi v1.0.2
      │   │   └── libdd-remote-config v3.0.0
      │   │       ├── libdd-ffe v1.0.0 (*)
      │   │       └── (dev) libdd-remote-config v3.0.0 (*)
      │   ├── libdd-common-ffi v40.0.0
      │   │   └── libdd-ffe-ffi v1.0.2 (*)
      │   ├── libdd-ffe v1.0.0 (*)
      │   └── libdd-remote-config v3.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-common v5.1.1 (*)

error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:260:1
    │
260 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── libdd-remote-config v3.0.0
          ├── libdd-ffe v1.0.0
          │   └── libdd-ffe-ffi v1.0.2
          └── (dev) libdd-remote-config v3.0.0 (*)

advisories FAILED, bans ok, sources ok

📦 `libdd-ffe-test-suite` - 4 error(s)

Show output

error[unmaintained]: Bincode is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:24:1
   │
24 │ bincode 1.3.3 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2025-0141
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2025-0141
   ├ Due to a doxxing and harassment incident, the bincode team has taken the decision to cease development permanently.
     
     The team considers version 1.3.3 a complete version of bincode that is not in need of any updates.
     
     ## Alternatives to consider
     
     * [wincode](https://crates.io/crates/wincode)
     * [postcard](https://crates.io/crates/postcard)
     * [bitcode](https://crates.io/crates/bitcode)
     * [rkyv](https://crates.io/crates/rkyv)
   ├ Announcement: https://git.sr.ht/~stygianentity/bincode/tree/v3.0/item/README.md
   ├ Solution: No safe upgrade is available!
   ├ bincode v1.3.3
     └── (dev) libdd-ffe v1.0.0
         └── (dev) libdd-ffe-test-suite v40.0.0

error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:55:1
   │
55 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     └── (dev) libdd-ffe-test-suite v40.0.0

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:185:1
    │
185 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── (dev) libdd-common v5.1.1
      │   ├── libdd-capabilities-impl v4.0.0
      │   │   ├── (dev) libdd-ffe v1.0.0
      │   │   │   └── (dev) libdd-ffe-test-suite v40.0.0
      │   │   └── libdd-remote-config v3.0.0
      │   │       ├── libdd-ffe v1.0.0 (*)
      │   │       └── (dev) libdd-remote-config v3.0.0 (*)
      │   ├── libdd-ffe v1.0.0 (*)
      │   └── libdd-remote-config v3.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-common v5.1.1 (*)

error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:249:1
    │
249 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── libdd-remote-config v3.0.0
          ├── libdd-ffe v1.0.0
          │   └── (dev) libdd-ffe-test-suite v40.0.0
          └── (dev) libdd-remote-config v3.0.0 (*)

advisories FAILED, bans ok, sources ok

📦 `libdd-ffe` - 4 error(s)

Show output

error[unmaintained]: Bincode is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:22:1
   │
22 │ bincode 1.3.3 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2025-0141
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2025-0141
   ├ Due to a doxxing and harassment incident, the bincode team has taken the decision to cease development permanently.
     
     The team considers version 1.3.3 a complete version of bincode that is not in need of any updates.
     
     ## Alternatives to consider
     
     * [wincode](https://crates.io/crates/wincode)
     * [postcard](https://crates.io/crates/postcard)
     * [bitcode](https://crates.io/crates/bitcode)
     * [rkyv](https://crates.io/crates/rkyv)
   ├ Announcement: https://git.sr.ht/~stygianentity/bincode/tree/v3.0/item/README.md
   ├ Solution: No safe upgrade is available!
   ├ bincode v1.3.3
     └── (dev) libdd-ffe v1.0.0

error[unsound]: Potential use-after-free due to lack of panic safety in `LruCache::pop()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:133:1
    │
133 │ lru 0.16.4 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0253
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0253
    ├ `LruCache::pop()` in `lru` was not panic-safe. If the `Drop` implementation of a stored key panics during `pop()`, `self.detach()` is never called, leaving dangling pointers in the internal doubly-linked list.
      
      A subsequent cache operation that triggers eviction can then dereference these dangling pointers:
      - The node is freed from the map, but remains linked in the LRU list due to the skipped `detach()` call
      - When a new insertion causes eviction, the LRU traversal encounters the dangling pointer
      - This results in a write to already-freed memory during the eviction process
      
      ## Impact
      
      - **CWE-416 (Use-After-Free):** memory corruption when subsequent cache operations access freed node pointers in the linked list
      - **CWE-415 (Double Free):** potential heap corruption when the same memory is freed multiple times
      
      Both types of undefined behavior can be invoked in safe Rust, but only if unwinding panics are enabled and `std::panic::catch_unwind` is used with key types that have potentially-panicking `Drop` implementations.
      
      ## Fix
      
      Fixed in `lru` 0.18.2 by detaching the node from the linked list before freeing it and dropping the key ([lru-rs#238](https://github.com/jeromefroe/lru-rs/pull/238)).
    ├ Announcement: https://github.com/jeromefroe/lru-rs/pull/238
    ├ Solution: Upgrade to >=0.18.2 (try `cargo update -p lru`)
    ├ lru v0.16.4
      └── libdd-ffe v1.0.0

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:164:1
    │
164 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── (dev) libdd-common v5.1.1
      │   ├── libdd-capabilities-impl v4.0.0
      │   │   ├── (dev) libdd-ffe v1.0.0
      │   │   └── libdd-remote-config v3.0.0
      │   │       ├── libdd-ffe v1.0.0 (*)
      │   │       └── (dev) libdd-remote-config v3.0.0 (*)
      │   ├── libdd-ffe v1.0.0 (*)
      │   └── libdd-remote-config v3.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-common v5.1.1 (*)

error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:225:1
    │
225 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── libdd-remote-config v3.0.0
          ├── libdd-ffe v1.0.0
          └── (dev) libdd-remote-config v3.0.0 (*)

advisories FAILED, bans ok, sources ok

Updated: 2026-08-11 16:39:02 UTC | Commit: 5bade8b | dependency-check job results

datadog-prod-us1-4 · 2026-08-11T16:41:37Z

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/apm-reliability/libdatadog | benchmarks 1/2

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a0f5226 | Docs | Datadog PR Page | Give us feedback!}

leoromanovsky · 2026-08-11T16:43:02Z

Superseded by #2339, which already implements the same canonical fixture bump, malformed-flag parse-error mapping, and SemVer precedence fix with green CI.

pr-commenter · 2026-08-11T16:53:27Z

Benchmarks

Comparison

Benchmark execution time: 2026-08-11 16:51:38

Comparing candidate commit a0f5226 in PR branch agent/isolate-malformed-flags with baseline commit 6ea3203 in branch main.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 13 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:sdk_test_data/rules-based

🟥 execution_time [+14.654µs; +16.857µs] or [+11.335%; +13.039%]

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`a0f5226`	1786466080	agent/isolate-malformed-flags

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
flagevaluation_evp/payloads/scale/2500flags_500users_20fields	execution_time	21.021ms	21.121ms ± 0.046ms	21.118ms ± 0.023ms	21.144ms	21.196ms	21.268ms	21.321ms	0.96%	0.881	2.547	0.22%	0.003ms	1	200
flagevaluation_evp/payloads/scale/2500flags_500users_20fields	throughput	117254.889op/s	118365.474op/s ± 256.238op/s	118381.403op/s ± 127.062op/s	118507.028op/s	118756.374op/s	118906.750op/s	118927.791op/s	0.46%	-0.857	2.475	0.22%	18.119op/s	1	200
flagevaluation_evp/payloads/stress/10flags_1000users_250fields	execution_time	95.266ms	95.987ms ± 0.359ms	95.983ms ± 0.239ms	96.191ms	96.513ms	96.738ms	98.338ms	2.45%	1.541	8.176	0.37%	0.025ms	1	200
flagevaluation_evp/payloads/stress/10flags_1000users_250fields	throughput	10169.025op/s	10418.276op/s ± 38.793op/s	10418.550op/s ± 25.991op/s	10445.874op/s	10473.171op/s	10489.204op/s	10496.978op/s	0.75%	-1.455	7.549	0.37%	2.743op/s	1	200
flagevaluation_evp/payloads/typical/100flags_50users_10fields	execution_time	578.451µs	580.477µs ± 0.840µs	580.255µs ± 0.366µs	580.690µs	582.326µs	583.066µs	583.441µs	0.55%	1.268	1.633	0.14%	0.059µs	1	200
flagevaluation_evp/payloads/typical/100flags_50users_10fields	throughput	171397.069op/s	172272.460op/s ± 248.924op/s	172337.960op/s ± 108.751op/s	172435.574op/s	172551.278op/s	172685.361op/s	172875.533op/s	0.31%	-1.260	1.613	0.14%	17.602op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
flagevaluation_evp/payloads/scale/2500flags_500users_20fields	execution_time	[21.115ms; 21.127ms] or [-0.030%; +0.030%]	None	None	None
flagevaluation_evp/payloads/scale/2500flags_500users_20fields	throughput	[118329.962op/s; 118400.986op/s] or [-0.030%; +0.030%]	None	None	None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields	execution_time	[95.937ms; 96.036ms] or [-0.052%; +0.052%]	None	None	None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields	throughput	[10412.899op/s; 10423.652op/s] or [-0.052%; +0.052%]	None	None	None
flagevaluation_evp/payloads/typical/100flags_50users_10fields	execution_time	[580.361µs; 580.593µs] or [-0.020%; +0.020%]	None	None	None
flagevaluation_evp/payloads/typical/100flags_50users_10fields	throughput	[172237.961op/s; 172306.958op/s] or [-0.020%; +0.020%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`a0f5226`	1786466080	agent/isolate-malformed-flags

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sdk_test_data/rules-based	execution_time	141.808µs	145.031µs ± 6.045µs	143.706µs ± 0.775µs	144.807µs	150.235µs	168.487µs	210.768µs	46.67%	7.814	73.947	4.16%	0.427µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sdk_test_data/rules-based	execution_time	[144.193µs; 145.869µs] or [-0.578%; +0.578%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`a0f5226`	1786466080	agent/isolate-malformed-flags

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields	execution_time	5.388ms	5.410ms ± 0.026ms	5.409ms ± 0.004ms	5.413ms	5.420ms	5.430ms	5.769ms	6.65%	12.524	167.297	0.49%	0.002ms	1	200
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields	throughput	433383.365op/s	462076.679op/s ± 2128.000op/s	462222.401op/s ± 373.055op/s	462583.372op/s	463176.769op/s	463701.723op/s	464005.644op/s	0.39%	-12.338	163.970	0.46%	150.472op/s	1	200
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields	execution_time	7.956ms	7.980ms ± 0.008ms	7.980ms ± 0.004ms	7.983ms	7.991ms	8.002ms	8.021ms	0.52%	0.543	4.214	0.10%	0.001ms	1	200
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields	throughput	124670.268op/s	125316.001op/s ± 129.228op/s	125320.698op/s ± 58.152op/s	125377.996op/s	125494.806op/s	125671.358op/s	125697.232op/s	0.30%	-0.524	4.165	0.10%	9.138op/s	1	200
flagevaluation_evp/coalescer/typical/100flags_50users_10fields	execution_time	173.915µs	186.383µs ± 39.592µs	183.886µs ± 0.152µs	184.043µs	184.288µs	184.635µs	743.124µs	304.12%	14.007	194.472	21.19%	2.800µs	1	200
flagevaluation_evp/coalescer/typical/100flags_50users_10fields	throughput	134567.044op/s	542690.181op/s ± 29350.826op/s	543815.910op/s ± 450.129op/s	544235.363op/s	550368.342op/s	573654.896op/s	574994.358op/s	5.73%	-13.519	185.827	5.39%	2075.417op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields	execution_time	[5.407ms; 5.414ms] or [-0.068%; +0.068%]	None	None	None
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields	throughput	[461781.759op/s; 462371.599op/s] or [-0.064%; +0.064%]	None	None	None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields	execution_time	[7.979ms; 7.981ms] or [-0.014%; +0.014%]	None	None	None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields	throughput	[125298.091op/s; 125333.910op/s] or [-0.014%; +0.014%]	None	None	None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields	execution_time	[180.896µs; 191.870µs] or [-2.944%; +2.944%]	None	None	None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields	throughput	[538622.439op/s; 546757.923op/s] or [-0.750%; +0.750%]	None	None	None

Group 4

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`a0f5226`	1786466080	agent/isolate-malformed-flags

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
single_flag_killswitch/rules-based	execution_time	156.239ns	159.527ns ± 2.465ns	159.226ns ± 1.604ns	160.848ns	163.925ns	166.967ns	169.027ns	6.16%	0.935	0.902	1.54%	0.174ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
single_flag_killswitch/rules-based	execution_time	[159.186ns; 159.869ns] or [-0.214%; +0.214%]	None	None	None

Baseline

Baseline benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`6ea3203`	1786449971	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
flagevaluation_evp/payloads/scale/2500flags_500users_20fields	execution_time	21.119ms	21.195ms ± 0.045ms	21.189ms ± 0.026ms	21.216ms	21.283ms	21.324ms	21.342ms	0.72%	0.766	0.564	0.21%	0.003ms	1	200
flagevaluation_evp/payloads/scale/2500flags_500users_20fields	throughput	117139.640op/s	117951.253op/s ± 248.216op/s	117988.495op/s ± 147.590op/s	118119.938op/s	118310.434op/s	118370.196op/s	118378.843op/s	0.33%	-0.753	0.537	0.21%	17.552op/s	1	200
flagevaluation_evp/payloads/stress/10flags_1000users_250fields	execution_time	95.123ms	95.944ms ± 0.374ms	95.890ms ± 0.171ms	96.109ms	96.565ms	96.691ms	98.953ms	3.19%	2.748	19.565	0.39%	0.026ms	1	200
flagevaluation_evp/payloads/stress/10flags_1000users_250fields	throughput	10105.853op/s	10422.857op/s ± 40.265op/s	10428.600op/s ± 18.599op/s	10444.483op/s	10474.268op/s	10497.038op/s	10512.680op/s	0.81%	-2.589	17.999	0.39%	2.847op/s	1	200
flagevaluation_evp/payloads/typical/100flags_50users_10fields	execution_time	582.903µs	584.342µs ± 0.722µs	584.266µs ± 0.436µs	584.725µs	585.498µs	586.046µs	589.141µs	0.83%	1.728	8.917	0.12%	0.051µs	1	200
flagevaluation_evp/payloads/typical/100flags_50users_10fields	throughput	169738.701op/s	171132.907op/s ± 210.884op/s	171154.926op/s ± 127.543op/s	171269.604op/s	171403.730op/s	171502.113op/s	171555.228op/s	0.23%	-1.699	8.693	0.12%	14.912op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
flagevaluation_evp/payloads/scale/2500flags_500users_20fields	execution_time	[21.189ms; 21.201ms] or [-0.029%; +0.029%]	None	None	None
flagevaluation_evp/payloads/scale/2500flags_500users_20fields	throughput	[117916.853op/s; 117985.654op/s] or [-0.029%; +0.029%]	None	None	None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields	execution_time	[95.893ms; 95.996ms] or [-0.054%; +0.054%]	None	None	None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields	throughput	[10417.277op/s; 10428.438op/s] or [-0.054%; +0.054%]	None	None	None
flagevaluation_evp/payloads/typical/100flags_50users_10fields	execution_time	[584.242µs; 584.442µs] or [-0.017%; +0.017%]	None	None	None
flagevaluation_evp/payloads/typical/100flags_50users_10fields	throughput	[171103.680op/s; 171162.133op/s] or [-0.017%; +0.017%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`6ea3203`	1786449971	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sdk_test_data/rules-based	execution_time	126.239µs	129.276µs ± 5.157µs	128.344µs ± 0.726µs	129.050µs	133.780µs	148.560µs	184.764µs	43.96%	7.641	71.206	3.98%	0.365µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sdk_test_data/rules-based	execution_time	[128.561µs; 129.991µs] or [-0.553%; +0.553%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`6ea3203`	1786449971	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields	execution_time	5.385ms	5.408ms ± 0.008ms	5.408ms ± 0.005ms	5.413ms	5.421ms	5.432ms	5.450ms	0.78%	0.891	2.917	0.15%	0.001ms	1	200
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields	throughput	458736.555op/s	462254.346op/s ± 713.113op/s	462317.527op/s ± 410.365op/s	462714.532op/s	463222.417op/s	463767.952op/s	464284.260op/s	0.43%	-0.872	2.849	0.15%	50.425op/s	1	200
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields	execution_time	7.871ms	7.901ms ± 0.013ms	7.900ms ± 0.005ms	7.905ms	7.917ms	7.941ms	8.000ms	1.27%	2.760	19.572	0.16%	0.001ms	1	200
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields	throughput	125004.281op/s	126560.949op/s ± 200.077op/s	126588.440op/s ± 73.179op/s	126646.507op/s	126768.493op/s	127022.225op/s	127051.242op/s	0.37%	-2.694	19.006	0.16%	14.148op/s	1	200
flagevaluation_evp/coalescer/typical/100flags_50users_10fields	execution_time	179.543µs	183.049µs ± 0.523µs	182.986µs ± 0.158µs	183.181µs	183.881µs	184.421µs	184.886µs	1.04%	-1.619	14.372	0.29%	0.037µs	1	200
flagevaluation_evp/coalescer/typical/100flags_50users_10fields	throughput	540874.961op/s	546305.227op/s ± 1568.628op/s	546489.462op/s ± 472.403op/s	546927.441op/s	547363.110op/s	553368.409op/s	556970.415op/s	1.92%	1.738	15.006	0.29%	110.919op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields	execution_time	[5.407ms; 5.409ms] or [-0.021%; +0.021%]	None	None	None
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields	throughput	[462155.515op/s; 462353.177op/s] or [-0.021%; +0.021%]	None	None	None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields	execution_time	[7.900ms; 7.903ms] or [-0.022%; +0.022%]	None	None	None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields	throughput	[126533.221op/s; 126588.678op/s] or [-0.022%; +0.022%]	None	None	None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields	execution_time	[182.977µs; 183.122µs] or [-0.040%; +0.040%]	None	None	None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields	throughput	[546087.830op/s; 546522.623op/s] or [-0.040%; +0.040%]	None	None	None

Group 4

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`6ea3203`	1786449971	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
single_flag_killswitch/rules-based	execution_time	155.705ns	159.361ns ± 2.775ns	158.874ns ± 1.585ns	160.453ns	164.390ns	168.887ns	171.814ns	8.14%	1.445	3.108	1.74%	0.196ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
single_flag_killswitch/rules-based	execution_time	[158.976ns; 159.746ns] or [-0.241%; +0.241%]	None	None	None

fix(ffe): report malformed flags as parse errors

a0f5226

leoromanovsky mentioned this pull request Aug 11, 2026

test(openfeature): cover isolated malformed flags DataDog/dd-trace-rb#6179

Draft

leoromanovsky mentioned this pull request Aug 11, 2026

fix(openfeature): consume isolated malformed flags DataDog/dd-trace-py#19626

Closed

leoromanovsky closed this Aug 11, 2026

Navigation Menu

Uh oh!

fix(ffe): report malformed flags as parse errors - #2343

fix(ffe): report malformed flags as parse errors#2343
leoromanovsky wants to merge 1 commit into
mainfrom
agent/isolate-malformed-flags

leoromanovsky commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

datadog-prod-us1-4 Bot commented Aug 11, 2026

Uh oh!

leoromanovsky commented Aug 11, 2026

Uh oh!

pr-commenter Bot commented Aug 11, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Group 1

Group 2

Group 3

Group 4

Group 1

Group 2

Group 3

Group 4

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

leoromanovsky commented Aug 11, 2026

Motivation

Changes

Decisions

Validation

Uh oh!

github-actions Bot commented Aug 11, 2026

📚 Documentation Check Results

📦 libdd-ffe-ffi - 545 warning(s)

📦 libdd-ffe-test-suite - ✅ No warnings

📦 libdd-ffe - 384 warning(s)

Uh oh!

github-actions Bot commented Aug 11, 2026

🔒 Cargo Deny Results

📦 libdd-ffe-ffi - 3 error(s)

📦 libdd-ffe-test-suite - 4 error(s)

📦 libdd-ffe - 4 error(s)

Uh oh!

datadog-prod-us1-4 Bot commented Aug 11, 2026

⚠️ Warnings

Uh oh!

leoromanovsky commented Aug 11, 2026

Uh oh!

pr-commenter Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Explanation

More details about the CI and significant changes

scenario:sdk_test_data/rules-based

Candidate

Group 1

Group 2

Group 3

Group 4

Baseline

Group 1

Group 2

Group 3

Group 4

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

📦 `libdd-ffe-ffi` - 545 warning(s)

📦 `libdd-ffe-test-suite` - ✅ No warnings

📦 `libdd-ffe` - 384 warning(s)

📦 `libdd-ffe-ffi` - 3 error(s)

📦 `libdd-ffe-test-suite` - 4 error(s)

📦 `libdd-ffe` - 4 error(s)

pr-commenter Bot commented Aug 11, 2026 •

edited

Loading