Harden telemetry delivery and refine recipe metadata

README.md

@@ -115,7 +115,8 @@ The sample chat app to run is found as [model-chat.py](https://github.com/micros
 - [Recipes](https://github.com/microsoft/olive-recipes)
 
 ## Data/Telemetry
-Distributions of this project may collect usage data and send it to Microsoft to help improve our products and services. See the [privacy statement](docs/Privacy.md) for more details.
+
+This project may collect usage data and send it to Microsoft to help improve our products and services. See the [privacy statement](docs/Privacy.md) for details.
 
 ## 🤝 Contributions and Feedback
 - We welcome contributions! Please read the [contribution guidelines](./CONTRIBUTING.md) for more details on how to contribute to the Olive project.
@@ -133,4 +134,3 @@ Licensed under the [MIT](./LICENSE) License.
 [![Build Status](https://dev.azure.com/aiinfra/PublicPackages/_apis/build/status%2FOlive%20CI?label=Olive-CI)](https://dev.azure.com/aiinfra/PublicPackages/_build/latest?definitionId=1240)
 [![Build Status](https://dev.azure.com/aiinfra/PublicPackages/_apis/build/status%2FOlive-ORT-Nightly?label=Olive-ORT-Nightly)](https://dev.azure.com/aiinfra/PublicPackages/_build/latest?definitionId=1279)
 
-

docs/Privacy.md

@@ -6,11 +6,15 @@ The software may collect information about you and your use of the software and
 ***
 
 ## Technical Details
-Olive uses the [OpenTelemetry](https://opentelemetry.io/) API for its implementation. Telemetry is turned ON by default. Based on user consent, this data may be periodically sent to Microsoft servers following GDPR and privacy regulations for anonymity and data access controls. Application, device, and version information is collected automatically.
+Telemetry is turned ON by default. Based on user consent, this data may be periodically sent to Microsoft servers following GDPR and privacy regulations for anonymity and data access controls. Application, device, and version information is collected automatically.
 
 In addition, Olive may collect additional telemetry data such as:
 - Invoked commands
 - Performance data
 - Exception information
 
-Collection of this additional telemetry can be disabled by adding the `--disable_telemetry` flag to any Olive CLI command, or by setting the `OLIVE_DISABLE_TELEMETRY` environment variable to `1` before running. Telemetry is also automatically disabled when a CI/CD environment is detected (e.g., GitHub Actions, Azure Pipelines, Jenkins). If telemetry is enabled, but cannot be sent to Microsoft, it will be stored locally and sent when a connection is available. You can override the default cache location by setting the `OLIVE_TELEMETRY_CACHE_DIR` environment variable to a valid directory path.
+You can disable all telemetry by adding the `--disable_telemetry` flag to any Olive CLI command, or by setting the `ORT_DISABLE_TELEMETRY` environment variable to `1` before running. This full opt-out applies for the lifetime of the process.
+
+In CI/CD environments (e.g., GitHub Actions, Azure Pipelines, Jenkins), Olive suppresses the device-id heartbeat and the action/error events and only emits the `OliveRecipe` event. The `OliveRecipe` event may include recipe metadata such as pass types, explicitly configured target settings, the host system type (including the default `LocalSystem` host) and any explicitly configured host accelerator settings, whether a custom package config was provided, a redacted snapshot of custom package-config overrides, and a redacted snapshot of explicitly supplied config overrides. Setting `ORT_DISABLE_TELEMETRY=1` in a CI/CD environment sends nothing at all.
+
+Telemetry is implemented using only the Python standard library. Events are written to a local per-user SQLite queue and uploaded in the background to Microsoft over HTTPS. If telemetry is enabled but cannot be sent (for example, while offline), events remain in the local queue and are uploaded on a later run when a connection is available.

olive/cli/base.py

@@ -338,7 +338,9 @@ def _run_workflow(self):
                     mark_test_output_path(self.args.output_path)
                 print("Dry run mode enabled. Configuration file is generated but no optimization is performed.")
                 return None
-            workflow_output = olive_run(run_config)
+            workflow_output = olive_run(
+                run_config, recipe_telemetry_metadata=self._get_recipe_telemetry_metadata(), emit_error_telemetry=False
+            )
             if is_test:
                 mark_test_output_path(self.args.output_path)
                 save_discrepancy_check_results(workflow_output, self.args.output_path)
@@ -349,6 +351,17 @@ def _run_workflow(self):
                 print(f"Model is saved at {self.args.output_path}")
             return workflow_output
 
+    def _get_recipe_telemetry_metadata(self) -> dict[str, str]:
+        recipe_name = self.__class__.__name__
+        if recipe_name.endswith("Command"):
+            recipe_name = recipe_name[: -len("Command")]
+        return {
+            "recipe_name": recipe_name,
+            "recipe_command": recipe_name,
+            "recipe_source": "generated_cli",
+            "recipe_format": "generated",
+        }
+
     @staticmethod
     def _parse_extra_options(kv_items):
         from onnxruntime_genai import __version__ as OrtGenaiVersion
@@ -995,7 +1008,11 @@ def add_search_options(sub_parser: ArgumentParser):
 
 def add_telemetry_options(sub_parser: ArgumentParser):
     """Add telemetry options to the sub_parser."""
-    sub_parser.add_argument("--disable_telemetry", action="store_true", help="Disable telemetry for this command.")
+    sub_parser.add_argument(
+        "--disable_telemetry",
+        action="store_true",
+        help="Disable all telemetry for this process.",
+    )
     return sub_parser
 
 

olive/cli/launcher.py

@@ -2,6 +2,7 @@
 # Copyright (c) Microsoft Corporation. All rights reserved.
 # Licensed under the MIT License.
 # --------------------------------------------------------------------------
+import os
 import sys
 from argparse import ArgumentParser
 from warnings import warn
@@ -66,18 +67,29 @@ def main(raw_args=None, called_as_console_script: bool = True):
 
     args, unknown_args = parser.parse_known_args(raw_args)
 
-    telemetry = Telemetry()
-    if args.disable_telemetry:
-        telemetry.disable_telemetry()
-
     if not hasattr(args, "func"):
         parser.print_help()
         sys.exit(1)
 
-    # Run the command
-    service = args.func(parser, args, unknown_args)
-    service.run()
-    telemetry.shutdown()
+    # Honor --disable_telemetry before constructing Telemetry so the process creates
+    # no telemetry resources and never drains queued events.
+    disable_telemetry = getattr(args, "disable_telemetry", False)
+    previous_opt_out = os.environ.get("ORT_DISABLE_TELEMETRY")
+    if disable_telemetry:
+        os.environ["ORT_DISABLE_TELEMETRY"] = "1"
+    telemetry = None
+    try:
+        telemetry = Telemetry()
+        service = args.func(parser, args, unknown_args)
+        service.run()
+    finally:
+        if telemetry is not None:
+            telemetry.shutdown()
+        if disable_telemetry:
+            if previous_opt_out is None:
+                os.environ.pop("ORT_DISABLE_TELEMETRY", None)
+            else:
+                os.environ["ORT_DISABLE_TELEMETRY"] = previous_opt_out
 
 
 def legacy_call(deprecated_module: str, command_name: str, *args):

olive/cli/run.py

@@ -55,16 +55,22 @@ def register_subcommand(parser: ArgumentParser):
 
     @action
     def run(self):
+        from copy import deepcopy
+        from pathlib import Path
+
         from olive.common.config_utils import load_config_file
         from olive.workflows import run as olive_run
 
         # allow the run_config to be a dict already (for api use)
-        run_config = self.args.run_config
-        if not isinstance(run_config, dict):
-            run_config = load_config_file(run_config)
+        run_config_input = self.args.run_config
+        run_config = (
+            deepcopy(run_config_input) if isinstance(run_config_input, dict) else load_config_file(run_config_input)
+        )
+        config_overrides = {}
         if input_model_config := get_input_model_config(self.args, required=False):
             print("Replacing input model config in run config")
             run_config["input_model"] = input_model_config
+            config_overrides["input_model"] = input_model_config
         elif self.args.test not in (None, False):
             input_model = run_config.get("input_model")
             if not isinstance(input_model, dict) or input_model.get("type", "").lower() != "hfmodel":
@@ -84,13 +90,28 @@ def run(self):
                 run_config.get("engine", {}).pop(rc_key, None)
                 # add value to run config directly
                 run_config[rc_key] = arg_value
+                config_overrides[rc_key] = arg_value
+
+        recipe_telemetry_metadata = {
+            "recipe_command": "WorkflowRun",
+            "recipe_source": "config_dict" if isinstance(run_config_input, dict) else "config_file",
+            "recipe_format": "dict"
+            if isinstance(run_config_input, dict)
+            else Path(run_config_input).suffix.lstrip(".").lower() or "unknown",
+            "execution_mode": "list_required_packages" if self.args.list_required_packages else "run",
+            "package_config_provided": bool(self.args.package_config),
+        }
+        if config_overrides:
+            recipe_telemetry_metadata["config_overrides"] = config_overrides
 
         output_path = run_config.get("output_dir") or run_config.get("engine", {}).get("output_dir")
         workflow_output = olive_run(
             run_config,
             list_required_packages=self.args.list_required_packages,
             tempdir=self.args.tempdir,
             package_config=self.args.package_config,
+            recipe_telemetry_metadata=recipe_telemetry_metadata,
+            emit_error_telemetry=False,
         )
         if self.args.test not in (None, False):
             mark_test_output_path(output_path)

olive/systems/docker/docker_system.py

@@ -232,6 +232,8 @@ def _prepare_run_params(self) -> dict:
 
     def _prepare_environment(self, base_env) -> dict:
         """Prepare environment variables for container."""
+        from olive.telemetry.telemetry import is_ci_environment
+
         # Convert list to dict if needed
         if isinstance(base_env, list):
             environment = {env.split("=")[0]: env.split("=")[1] for env in base_env}
@@ -241,6 +243,8 @@ def _prepare_environment(self, base_env) -> dict:
         # Add default environment variables
         environment.setdefault("PYTHONPYCACHEPREFIX", "/tmp")
         environment["OLIVE_LOG_LEVEL"] = logging.getLevelName(logger.getEffectiveLevel())
+        if is_ci_environment():
+            environment["CI"] = "1"
 
         # Add HuggingFace token if needed
         if self.hf_token:

olive/systems/docker/workflow_runner.py

@@ -5,6 +5,7 @@
 
 from olive.common.hf.login import huggingface_login
 from olive.logging import get_olive_logger, set_verbosity_from_env
+from olive.telemetry.telemetry import Telemetry
 from olive.workflows import run as olive_run
 
 logger = get_olive_logger()
@@ -20,7 +21,16 @@ def runner_entry(config):
         config = json.load(f)
 
     logger.info("Running workflow with config: %s", config)
-    olive_run(config)
+    try:
+        olive_run(config, emit_error_telemetry=False, emit_recipe_telemetry=False)
+    finally:
+        telemetry = Telemetry.get_existing_instance()
+        if telemetry is not None:
+            telemetry.shutdown(
+                timeout_millis=15_000,
+                callback_timeout_millis=15_000,
+                flush_seconds=15,
+            )
 
 
 if __name__ == "__main__":

olive/telemetry/deviceid/__init__.py

@@ -1,3 +1,3 @@
-from olive.telemetry.deviceid.deviceid import get_encrypted_device_id_and_status
+from olive.telemetry.deviceid.deviceid import get_hashed_device_id_and_status
 
-__all__ = ["get_encrypted_device_id_and_status"]
+__all__ = ["get_hashed_device_id_and_status"]

olive/telemetry/deviceid/_store.py

@@ -6,6 +6,14 @@
 REGISTRY_KEY = "deviceid"
 
 
+def _chmod_best_effort(path: Path, mode: int) -> None:
+    try:
+        path.chmod(mode)
+    except OSError:
+        # Permission tightening is best-effort on filesystems that do not support chmod.
+        pass
+
+
 class Store:
     def __init__(self) -> None:
         self._file_path: Path = self._build_path
@@ -21,9 +29,8 @@ def retrieve_id(self) -> str:
         :return: The device id.
         :rtype: str
         """
-        # check if file doesnt exist and raise an Exception
         if not self._file_path.is_file():
-            raise FileExistsError(f"File {self._file_path.stem} does not exist")
+            raise FileNotFoundError(f"File {self._file_path.stem} does not exist")
 
         return self._file_path.read_text(encoding="utf-8").strip()
 
@@ -33,10 +40,16 @@ def store_id(self, device_id: str) -> None:
         :param str device_id: The device id to store.
         :type device_id: str
         """
-        # create the folder location if it does not exist
-        self._file_path.parent.mkdir(parents=True, exist_ok=True)
-
-        self._file_path.touch()
+        # create the folder location if it does not exist, owner-only (0700) so other users on the
+        # machine cannot traverse into it to reach the device id.
+        self._file_path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
+        _chmod_best_effort(self._file_path.parent, 0o700)
+
+        # Owner-only (0600): the device id must not be world-readable by other users on the machine.
+        # touch(mode=...) creates it already restricted; chmod also tightens a pre-existing file before
+        # writing, so the id is never left at the umask default (commonly world-readable 0644).
+        self._file_path.touch(mode=0o600)
+        _chmod_best_effort(self._file_path, 0o600)
         self._file_path.write_text(device_id, encoding="utf-8")
 
 
@@ -65,6 +78,6 @@ def store_id(self, device_id: str) -> None:
             winreg.HKEY_CURRENT_USER,
             REGISTRY_PATH,
             reserved=0,
-            access=winreg.KEY_ALL_ACCESS | winreg.KEY_WOW64_64KEY,
+            access=winreg.KEY_SET_VALUE | winreg.KEY_CREATE_SUB_KEY | winreg.KEY_WOW64_64KEY,
         ) as key_handle:
             winreg.SetValueEx(key_handle, REGISTRY_KEY, 0, winreg.REG_SZ, device_id)

olive/telemetry/deviceid/deviceid.py

@@ -15,6 +15,7 @@ class DeviceIdStatus(Enum):
 
 
 _device_id_state = {"device_id": None, "status": DeviceIdStatus.NEW}
+_DEVICE_ID_HASH_SALT = "olive:"
 
 
 def get_device_id() -> str:
@@ -86,16 +87,8 @@ def get_device_id() -> str:
     return device_id
 
 
-def get_encrypted_device_id_and_status() -> tuple[str, DeviceIdStatus]:
-    """Generate a FIPS-compliant encrypted device ID using SHA256 and returns the deviceIdStatus.
-
-    This method uses SHA256 which is FIPS 140-2 approved for cryptographic operations.
-    The device ID is hashed to ensure deterministic but secure device identification.
-
-    Returns:
-        str: FIPS-compliant encrypted device ID (base64-encoded)
-
-    """
+def get_hashed_device_id_and_status() -> tuple[str, DeviceIdStatus]:
+    """Get the product-salted hashed device ID and its status."""
     device_id = _device_id_state["device_id"] if _device_id_state["device_id"] is not None else get_device_id()
-    encrypted_device_id = hashlib.sha256(device_id.encode("utf-8")).digest().hex().upper() if device_id else ""
-    return encrypted_device_id, _device_id_state["status"]
+    hashed = hashlib.sha256(f"{_DEVICE_ID_HASH_SALT}{device_id}".encode()).hexdigest() if device_id else ""
+    return f"c:{hashed}" if hashed else "", _device_id_state["status"]