Harden telemetry delivery and refine recipe metadata

[bmehta001]

Same-repository replacement for #2441 so Azure CI runs with the normal microsoft/Olive PR secret context.

Summary

• harden Olive telemetry delivery and cache behavior for non-CI runs, including Windows file locking, cache-path handling, and unreadable flush preservation
• keep CI telemetry ephemeral: CI emits only OliveRecipe and does not register local cache callbacks
• prevent Docker workflows from double-counting OliveRecipe by suppressing the inner container recipe event and forwarding CI detection into workflow containers
• refine OliveRecipe metadata so target fields are emitted only for explicit targets, host fields are tracked separately, and recipe/config telemetry captures redacted explicit overrides
• replace opaque package-config hashing with redacted package_config_overrides, while keeping package_config out of recipe_hash
• make recipe source/hash metadata deterministic by avoiding filesystem-dependent model/resource classification
• add or update focused telemetry, workflow, and Docker tests and refresh the privacy documentation

Why

We want telemetry to be reliable under cache contention and offline retry, while keeping CI runs temporal and recipe-only. We also want the recipe signal to be useful for analyzing explicit overrides without mixing host/target semantics, relying on opaque hashes, or depending on local filesystem state.

Validation

• lintrunner f -a -m main
• python -m compileall olive\\telemetry\\constants.py olive\\telemetry\\telemetry.py olive\\workflows\\run\\run.py olive\\systems\\docker\\docker_system.py test\\test_telemetry.py test\\workflows\\test_workflow_run.py test\\systems\\docker\\test_docker_system.py
• pytest test\\test_telemetry.py test\\workflows\\test_workflow_run.py test\\systems\\docker\\test_docker_system.py -q
• confirmed .azure_pipelines and .github do not set OLIVE_DISABLE_TELEMETRY, disable_telemetry, or --disable_telemetry

[Copilot]

Pull request overview

This PR extends Olive's telemetry pipeline so workflow runs emit a new OliveRecipe signal with redacted recipe metadata, while also tightening cache handling and documenting the updated telemetry behavior. It fits into the workflow, CLI, and telemetry layers that capture how Olive runs recipes across local and CI environments.

Changes:

• Adds recipe-result telemetry from workflow execution and CLI entry points, including recipe hashes, model/source metadata, host/target metadata, and redacted config/package override snapshots.
• Reworks telemetry delivery details, including cache-path overrides, JSON-line cache storage, Windows file locking, unreadable flush preservation, and CI recipe-only filtering.
• Adds focused workflow and telemetry tests, plus privacy-documentation updates.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/workflows/test_workflow_run.py	Adds coverage for recipe-result metadata emitted by workflow runs.
test/test_telemetry.py	Adds tests for cache-path overrides, CI filtering, unreadable flush handling, and Windows file locking.
olive/workflows/run/run.py	Builds and emits recipe telemetry metadata during workflow execution.
olive/telemetry/utils.py	Reworks cross-platform exclusive file locking for telemetry cache access.
olive/telemetry/telemetry.py	Adds OliveRecipe, CI filtering, and cache/flush behavior changes.
olive/telemetry/telemetry_extensions.py	Adds a helper for logging recipe results.
olive/telemetry/library/telemetry_logger.py	Adds locking around singleton logger setup and configurable service name.
olive/telemetry/library/options.py	Extends exporter options with service_name.
olive/telemetry/constants.py	Updates the encoded telemetry connection string.
olive/cli/run.py	Passes workflow-run recipe metadata into telemetry.
olive/cli/base.py	Passes generated-CLI recipe metadata from higher-level commands.
docs/Privacy.md	Updates the privacy text for the new recipe telemetry behavior.

[bmehta001]

/AzurePipelines help

[azure-pipelines]

Supported commands

• help:
• Get descriptions, examples and documentation about supported commands
• Example: help "command_name"
• list:
• List all pipelines for this repository using a comment.
• Example: "list"
• run:
• Run all pipelines or specific pipelines for this repository using a comment. Use this command by itself to trigger all related pipelines, or specify specific pipelines to run.
• Example: "run" or "run pipeline_name, pipeline_name, pipeline_name"
• where:
• Report back the Azure DevOps orgs that are related to this repository and org
• Example: "where"

See additional documentation.

[bmehta001]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[shaahji]

• Have you considered using a local db solution like sqllite3 or tinydb for storing data? These might have a lot of added benefits compared to holding information in a file on disk.
• Olive config paths can contain pathlib.Path values. Default json package doesn't like these when invoked with calls to json.dump or json.dumps. You might want to check for those.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

olive/cli/launcher.py:87

• The ORT_DISABLE_TELEMETRY environment variable is restored immediately after constructing Telemetry(), before the command actually runs. This means --disable_telemetry may not suppress downstream telemetry (e.g., ONNX Runtime) during service.run(), even though the flag name and privacy docs imply it should apply for the whole invocation. Consider keeping the env var set until after the command completes (and telemetry is shut down), then restoring it to avoid leaking into subsequent invocations in the same process.

        if disable_telemetry:
            if previous_opt_out is None:
                os.environ.pop("ORT_DISABLE_TELEMETRY", None)
            else:
                os.environ["ORT_DISABLE_TELEMETRY"] = previous_opt_out

[Copilot]

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

olive/telemetry/telemetry.py:214

• PR description says CI telemetry should be ephemeral, but the CI path still initializes a durable SQLite offline store under the normal per-user telemetry base directory. On hosted CI agents where the cache directory persists between jobs, this can carry over OliveRecipe rows across unrelated runs and defeat the “ephemeral CI” goal.

Consider making CI recipe-only telemetry truly ephemeral (e.g., bypass OfflineEventStore and POST directly, or use a per-run temp directory / delete the DB on shutdown) so CI never writes durable telemetry state to the shared cache location.

                # Durable on-disk queue + background uploader. The uploader
                # retries enabled-run events until delivery.
                db_path = os.path.join(get_telemetry_base_dir(), DB_FILE_NAME)
                self._store = OfflineEventStore(db_path)
                if not self._store.is_open:

[Copilot]

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

olive/workflows/run/run.py:224

• _build_recipe_result_metadata computes config_overrides from the run_config_telemetry_input parameter when the caller doesn’t explicitly provide overrides. Here run() always passes None, so recipe telemetry will never auto-capture redacted config overrides from a config file/dict (contrary to the intended behavior).

                metadata = _build_recipe_result_metadata(
                    run_config,
                    None,
                    parsed_run_config,
                    recipe_telemetry_metadata,

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated no new comments.