Skip to content

Allow telemetry-free OpenFeature providers - #12387

Open
leoromanovsky wants to merge 6 commits into
masterfrom
leo.romanovsky/named-client-telemetry
Open

Allow telemetry-free OpenFeature providers#12387
leoromanovsky wants to merge 6 commits into
masterfrom
leo.romanovsky/named-client-telemetry

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Adds a provider-level option that allows an OpenFeature domain to evaluate Datadog feature flags without emitting Datadog telemetry.

Motivation

A customer needs to evaluate the same flag in two modes from one application:

  • a live path that makes the real decision and records its exposure and flag-evaluation telemetry;
  • a peek path that inspects what value or variant a context would receive without recording an exposure or polluting evaluation telemetry.

Named clients alone share their domain's provider behavior. OpenFeature named domains let the application bind one Datadog provider to the live client and a separate telemetry-free Datadog provider to the peek client while both providers continue to consume the same Datadog flag configuration.

OpenFeatureAPI api = OpenFeatureAPI.getInstance();

api.setProviderAndWait("live", new Provider());
api.setProviderAndWait(
    "peek",
    new Provider(new Provider.Options().telemetryEnabled(false)));

Client checkoutClient = api.getClient("live");
Client analyticsClient = api.getClient("peek");

EvaluationContext checkoutContext = new MutableContext("session-abc");
EvaluationContext analyticsContext = new MutableContext("user-123");

// Evaluates normally and emits the configured Datadog telemetry.
boolean checkoutEnabled = checkoutClient.getBooleanValue(
    "my-feature", false, checkoutContext);

// Evaluates against the shared current configuration without emitting Datadog telemetry.
boolean analyticsEnabled = analyticsClient.getBooleanValue(
    "my-feature", false, analyticsContext);

Changes

  • Add Provider.Options.telemetryEnabled(boolean), defaulting to true.
  • Preserve the existing 30-second initialization default for every new Provider.Options; this also fixes new Provider(new Options()), which previously passed a null timeout unit into initialization.
  • Skip the existing EVP flag-evaluation, OpenTelemetry metric, and APM span-enrichment hooks and metadata when provider telemetry is disabled.
  • Pass the provider's immutable telemetry setting to the evaluator and guard the existing exposure dispatch with it.
  • Store one process-wide, versioned configuration snapshot in FeatureFlaggingGateway; evaluators read that shared snapshot once per evaluation, while listener callbacks only drive provider state events.
  • Cover reflective provider construction, shared snapshot races, one-poller activation, hook uniqueness, and exposure suppression with tests.

Decisions

  • Scope the option to a Provider, matching OpenFeature's domain-to-provider model. A single provider instance keeps one telemetry policy; applications that need both policies register two provider instances under separate domains.
  • Keep exposure dispatch in the evaluator and use a provider-scoped boolean. No shared mutable flag, mutex, thread-local, or new exposure hook is required.
  • Preserve existing behavior by enabling telemetry by default.
  • Snapshot the mutable Provider.Options values at provider construction so timeout, unit, and telemetry policy have one consistent binding point.
  • Require dd-openfeature 1.66.0 and later to run with dd-java-agent 1.66.0 or later; fail with a compatibility-focused error instead of falling back to per-evaluator configuration that can race.
  • Define telemetry-free to include exposures, EVP flag-evaluation events, OpenTelemetry evaluation metrics, and APM span enrichment. Flag resolution, configuration delivery, and returned evaluation details are unchanged.

Compatibility

dd-openfeature 1.66.0 and later requires dd-java-agent 1.66.0 or later because the shared configuration snapshot is provided by the agent bootstrap API. This is a minimum compatibility floor, not a lockstep requirement: provider and agent versions may differ once the agent meets that minimum. Falling back to the old per-evaluator state would reintroduce the configuration race this PR closes.

Configuration Downloads and Billing

The two providers do not create two configuration download streams within the same JVM. This is true for both supported delivery modes:

  • With Agent Remote Configuration, the process-wide Feature Flagging subsystem creates one RemoteConfigServiceImpl, which registers one FFE_FLAGS listener on the tracer's shared ConfigurationPoller.
  • With agentless delivery, the first provider activation consumes the process-wide activation listener and starts one AgentlessConfigurationSource polling loop. Later provider activations do not initialize another source.
  • In both modes, each provider owns an in-memory evaluator that registers with FeatureFlaggingGateway. A configuration received by the single source is cached by the gateway and fanned out to every evaluator; evaluating through both clients performs two local evaluations but no per-evaluation network request.

Datadog Feature Flags billing is based on Monthly Flag Configuration Requests, not individual evaluations or exposure events. Both Remote Configuration and agentless/CDN requests contribute to server-side Feature Flags billing, but adding the second provider in the same JVM does not add another poller and therefore does not add configuration requests or incremental MFCR usage.

This sharing guarantee is scoped to one JVM. Additional JVMs, processes, or containers can create additional configuration-delivery traffic; deployment-wide MFCR usage still depends on the deployment footprint and polling interval, with Agent Remote Configuration able to consolidate delivery through a shared Agent.

telemetryEnabled(false) suppresses the peek provider's Feature Flags exposure, evaluation metric, EVP flag-evaluation, and span-enrichment signals. It does not disable the shared configuration source or unrelated APM tracing of the surrounding application code.

Dogfooding Validation

Revalidated end to end on 2026-09-03 in the private ffe-dogfooding application against dd-trace-java head a7b19afdf9a0ddc9771655cb0ae0c1b1a369f875 and dogfooding main 91c8cf523809ccb4facf8bd54a6eb9cac69f1533 using this PR's locally built dd-openfeature and dd-java-agent artifacts. The full validation was then repeated with the immutable CI-produced dd-java-agent.jar, whose embedded dd-java-agent.version is 1.66.0-SNAPSHOT~a7b19afdf9; it produced the same PASS results. The private repository is intentionally not linked here; the measured evidence is included below.

The Java app registered live and peek OpenFeature domains, pointed agentless delivery at a deterministic request-counting UFC endpoint, and set the polling interval to 60 seconds. Both providers reached READY while the process reported exactly one live dd-feature-flagging-http-poller thread. The UFC endpoint recorded exactly one initial configuration request, rather than one request per provider.

After clearing the local exposure, evaluation, metric, and trace captures:

  • One live evaluation produced exactly one exposure, one EVP flag-evaluation row with evaluation_count: 1, one OpenTelemetry evaluation metric, and a root span carrying ffe_* enrichment tags.
  • One peek evaluation completed and flushed an ordinary application root span. After a 12-second settle window, exposure rows remained at 1, EVP rows remained at 1 with total evaluation_count: 1, and the OpenTelemetry metric count remained at 1. Zero spans from the peek request carried any ffe_* tag.

The reproducible validation completed with:

PASS configuration: two providers share 1 poller and produced 1 initial request
PASS live control: 1 exposure, 1 EVP evaluation, 1 OTLP evaluation, enriched root span
PASS peek: ordinary trace present, no span enrichment, and no exposure/EVP/OTLP increments
PASS Java telemetry-free named-domain validation

Additional Notes

Validated against master at a1b6599a5f9f8d803cfac89fa1c3befe41b8c33f with:

  • ./gradlew :products:feature-flagging:feature-flagging-bootstrap:test :products:feature-flagging:feature-flagging-api:test :products:feature-flagging:feature-flagging-lib:test :products:feature-flagging:feature-flagging-agent:test
  • DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true ./gradlew --no-build-cache :products:feature-flagging:feature-flagging-api:cleanTest :products:feature-flagging:feature-flagging-api:test
  • ./gradlew :products:feature-flagging:feature-flagging-api:spotlessCheck :products:feature-flagging:feature-flagging-api:javadoc

Contributor Checklist

  • Format the title according to the contribution guidelines.
  • Assign the required type: and comp: labels.
  • Keep the new behavior backward compatible by default.
  • Document the public option and usage example.

Allow OpenFeature domains to evaluate flags without producing Datadog telemetry.

Environment: Datadog workspace
@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Sep 2, 2026
Keep exposure dispatch in the evaluator and guard telemetry with a provider-scoped boolean.

Environment: Datadog workspace
@dd-octo-sts

dd-octo-sts Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.90 s 13.98 s [-1.4%; +0.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.87 s 12.98 s [-1.5%; -0.2%] (maybe better)
startup:petclinic:appsec:Agent 17.50 s 17.32 s [+0.2%; +1.9%] (maybe worse)
startup:petclinic:iast:Agent 17.29 s 17.53 s [-2.1%; -0.7%] (maybe better)
startup:petclinic:profiling:Agent 17.42 s 17.03 s [+1.0%; +3.5%] (significantly worse)
startup:petclinic:sca:Agent 17.41 s 16.75 s [-0.5%; +8.4%] (no difference)
startup:petclinic:tracing:Agent 16.11 s 16.66 s [-7.4%; +0.8%] (no difference)

Commit: ffdd4ff1 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation is backward-compatible and well tested; only a minor documentation wording correction remains.

Pull request overview

Adds provider-scoped suppression of OpenFeature telemetry while preserving flag evaluation behavior.

Changes:

  • Adds Options.telemetryEnabled(boolean), defaulting to enabled.
  • Suppresses telemetry hooks and exposure dispatch when disabled.
  • Adds tests and named-domain usage documentation.
File summaries
File Description
Provider.java Configures telemetry policy and hooks.
DDEvaluator.java Guards exposure dispatch.
ProviderTest.java Tests options and hook suppression.
DDEvaluatorTest.java Tests exposure suppression.
README.md Documents telemetry-free domains.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread products/feature-flagging/feature-flagging-api/README.md Outdated
@leoromanovsky
leoromanovsky marked this pull request as ready for review September 2, 2026 23:35
@leoromanovsky
leoromanovsky requested a review from a team as a code owner September 2, 2026 23:35
@leoromanovsky
leoromanovsky requested review from typotter and vjfridge and removed request for a team September 2, 2026 23:35
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-02T23:38:30.966816Z 43c2472 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The provider option stops exposure dispatch and all provider telemetry hooks. The default option keeps current telemetry behavior.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 43c2472 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

aarsilv commented Sep 3, 2026

Copy link
Copy Markdown

🤖 From Codex working with Aaron:

Ask: Use one process-wide, versioned configuration snapshot as the source for all providers. Keep the existing single poller and request path. Do not copy the configuration into each evaluator.

The current register-and-replay flow can apply old configuration A after new configuration B. This can leave one domain on A until a later update.

Publish one shared snapshot:

record ConfigSnapshot(long version, ServerConfiguration config) {}

static void dispatch(ServerConfiguration config) {
  ConfigSnapshot next =
      new ConfigSnapshot(VERSION.incrementAndGet(), config);
  CURRENT.set(next);
  notifyListeners(next);
}

Each DDEvaluator should read CURRENT once at the start of an evaluation. Listeners should control only provider state events. They should ignore a version that is not newer than the last version that they saw.

This gives the required rule: an evaluation before publication uses A, and an evaluation after publication uses B. It is acceptable if two separate evaluations are on opposite sides of an update and return different results.

A lock around registration and dispatch can prevent stale replay, but it still updates evaluators one at a time. The shared snapshot gives the stronger guarantee. The telemetry option stays local to each provider. Configuration requests stay shared for the JVM.

Please add tests for these cases:

  • An update during provider registration cannot restore an old configuration.
  • Two providers read the same current version.
  • An unavailable configuration becomes visible to all providers at once.
  • Two activations still create one agentless poller and one Remote Config listener.

@aarsilv aarsilv left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking care of this so quickly! I like how the shared downstream configuration requesting mechanisms avoid multiple config requests. Approving assuming you push the sharing down a level so there is config race condition (see comment from me working with Codex).

Comments from me working with Claude incoming, those you can take or leave as you see fit (but recommend taking, and I'm experimenting authoring them in a way they are hopefully easier to incorporate)

Comment thread products/feature-flagging/feature-flagging-api/README.md
@leoromanovsky

Copy link
Copy Markdown
Contributor Author

🤖 From Codex working with Aaron:

Ask: Use one process-wide, versioned configuration snapshot as the source for all providers. Keep the existing single poller and request path. Do not copy the configuration into each evaluator.

The current register-and-replay flow can apply old configuration A after new configuration B. This can leave one domain on A until a later update.

Publish one shared snapshot:

record ConfigSnapshot(long version, ServerConfiguration config) {}

static void dispatch(ServerConfiguration config) {
  ConfigSnapshot next =
      new ConfigSnapshot(VERSION.incrementAndGet(), config);
  CURRENT.set(next);
  notifyListeners(next);
}

Each DDEvaluator should read CURRENT once at the start of an evaluation. Listeners should control only provider state events. They should ignore a version that is not newer than the last version that they saw.

This gives the required rule: an evaluation before publication uses A, and an evaluation after publication uses B. It is acceptable if two separate evaluations are on opposite sides of an update and return different results.

A lock around registration and dispatch can prevent stale replay, but it still updates evaluators one at a time. The shared snapshot gives the stronger guarantee. The telemetry option stays local to each provider. Configuration requests stay shared for the JVM.

Please add tests for these cases:

  • An update during provider registration cannot restore an old configuration.
  • Two providers read the same current version.
  • An unavailable configuration becomes visible to all providers at once.
  • Two activations still create one agentless poller and one Remote Config listener.

@aarsilv Interesting find thank you - I'll start with a TDD to demonstrate whether this is happening before addressing it.

@leoromanovsky

Copy link
Copy Markdown
Contributor Author

avoid multiple config requests.

A hard requirement in my mind; I don't want to make customers pay twice for this.

Move the process-wide configuration snapshot into the gateway so providers cannot diverge during concurrent registration.

Environment: Datadog workspace
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 57.45% (-1.57%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ffdd4ff | Docs | View more details | Give us feedback!

Comment thread products/feature-flagging/feature-flagging-api/README.md
Cover telemetry suppression through reflective provider construction, snapshot provider options, and clarify agent compatibility failures.

Environment: Datadog workspace
Document the 1.66.0 agent API floor without implying lockstep provider and agent versions.

Environment: Datadog workspace
Expose the provider beside the agent in the existing public S3 snapshot job.

Environment: Datadog workspace
@leoromanovsky
leoromanovsky requested review from a team as code owners September 3, 2026 16:26
@leoromanovsky
leoromanovsky requested review from dougqh and randomanderson and removed request for a team September 3, 2026 16:26

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The provider option guards exposure, metric, event, and span enrichment paths. Each evaluator reads the shared versioned process snapshot.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit ffdd4ff · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing this issue so quickly. Long term, we need to get away from requiring dd-openfeature to be in lockstep version-wise with dd-java-agent.
That's one of the general platform requirements to work with single step installer.
But until then, this gives a clear diagnostic indication that will be helpful to support, so I'm approving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants