Skip to content

feat: Metrics tracking system for entanglement management #376

Open
Tw1ZZLER wants to merge 65 commits into
masterfrom
metrics
Open

feat: Metrics tracking system for entanglement management #376
Tw1ZZLER wants to merge 65 commits into
masterfrom
metrics

Conversation

@Tw1ZZLER

@Tw1ZZLER Tw1ZZLER commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Continuation of implementation of #368 and direct continuation of development after #374.

This PR implements a more generalized central metrics tracking system for SeQUeNCe. The module has been split up until multiple modules:

  • metrics/__init__.py: The front-facing API methods that are used to interact with the metrics module. Includes enable, record, configure, collect_trial_metrics, and aggregate_trial_metrics.
  • event_types.py: Defines methods for creating and handling event types. Event types are what is used by the record method to record a metric.
  • builtins.py: Built-in event types and metrics. Used for abstraction and categorizing the different types of metrics used throughout SeQUeNCe. Also gives an example of how a user could create their own metrics for their own experiments!
  • metrics.py: Mostly defines a Metric abstract class and the types of Metrics that are available. Each metric includes collect method that collects the information from those metrics, and this is typically used with the collect_trial_metrics method.
  • registry.py: Metric registry that provides convenient methods for registering and creating new metrics. All Metric types are stored as a list _metrics in this module.
  • storage.py: Mostly unchanged from MVP implementation of centralized metrics tracking system #374, just all storage methods have been moved to their own module, along with the the time provider classes.

Metric Superclass

Metrics have been generalized into certain metric types, that change the behavior of certain methods. Metric is now an abstract superclass of all the other metric types, which are implemented as data classes.

Abstract Properties

Metrics requires the abstract properties event_types and output_keys to be set:

  • event_types is an immutable set of the EventTypes that the metric reacts to during record.
  • output_keys is an immutable set of the keys that the collect() method produces in it's output dictionary. Useful for aggregation of data. collect() is discussed in the next section.

Collect Method

Metric also requires an abstract method in the form of collect(). The collect() method is to generalize the global collect_trial_metrics() method, which now just calls the collect() method of each metric and puts them all into one list. collect() is implemented differently and outputs different sets of information based on the metric.

  • CounterPairMetric collects the failures, successes, and success rate of the passed owner. This helps generalize the successes and failures that all the entanglement protocols produce.
  • LastValueMetric collects the last scalar field value from matching events. This helps generalize values like the throughput of an app.
  • EventFieldListMetric collects a list of field values matching a certain event type. This helps generalize giant lists of fidelities, like purified fidelities over time.
  • DeliveryTimeMetric collects the time to deliver N purified pairs relative to reservation start. This helps generalized values like purified delivery of the entanglement purification protocol (RequestApp, etc.).

On Record Method

on_record() is an optional method that is called when the metric is recorded. It is only used by CounterPairMetric, where it updates the internal success and failure counters and updates the success rate.

Collect Context

CollectContext is a simple data class that lays out a template context to pass to the collect() method of each metric. It also tells the collect() method what it can expect when it gets called. It is only used by the collect_trial_metrics() method. This need some more work to be properly generalized or at least easily extended by the user.

Collect Trial Metrics Method

The collect_trial_metrics method is now more generalized for all types of metrics. All it does now is iterate through the list of metrics and calls the collect() method of each metric. It uses the created CollectContext object with the passed arguments, to pass those arguments through to the metrics collect() methods. However, these arguments are currently specific to certain protocols and are not currently generalized.

Tw1ZZLER added 30 commits June 18, 2026 15:25
I don't know if 2 Timelines will ever exist at once, but if so then this
might be a problem. For now it seems to be an okay solution
@github-code-quality

github-code-quality Bot commented Jun 26, 2026

Copy link
Copy Markdown

Code Coverage Overview

Languages: Python

Python / code-coverage/pytest

The overall coverage in the branch remains at 83%, unchanged from the branch.

Show a code coverage summary of the most impacted files.
File e215984 b89ffb9 +/-
sequence/kernel...r/stabilizer.py 76% 76% 0%
sequence/components/bsm.py 82% 82% 0%
sequence/entang...n/bbpssw_bds.py 48% 48% 0%
sequence/utils/.../event_types.py 0% 81% +81%
sequence/utils/...ics/registry.py 0% 87% +87%
sequence/utils/...ics/__init__.py 0% 94% +94%
sequence/utils/...rics/storage.py 0% 96% +96%
sequence/utils/...metric_types.py 0% 98% +98%
tests/utils/test_metrics.py 0% 100% +100%
sequence/utils/...ics/builtins.py 0% 100% +100%

Updated July 01, 2026 17:26 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@Tw1ZZLER Tw1ZZLER linked an issue Jun 26, 2026 that may be closed by this pull request

@caitaozhan caitaozhan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments. Please take a look.

Comment thread sequence/entanglement_management/generation/generation_base.py Outdated
Comment thread sequence/utils/metrics/__init__.py Outdated
Comment thread sequence/utils/metrics/__init__.py Outdated
Comment thread sequence/utils/metrics/registry.py Outdated
Comment thread sequence/utils/metrics/metric_types.py Outdated
Comment thread sequence/utils/metrics/metric_types.py Outdated
Comment thread sequence/utils/metrics/metric_types.py Outdated
Comment thread sequence/utils/metrics/metric_types.py Outdated
Comment thread sequence/utils/metrics/metric_types.py Outdated
@hayekr hayekr changed the title Metrics tracking system for entanglement management feat: Metrics tracking system for entanglement management Jun 30, 2026
@hayekr hayekr requested a review from caitaozhan June 30, 2026 03:45
@hayekr hayekr marked this pull request as draft June 30, 2026 05:27
@hayekr hayekr marked this pull request as ready for review June 30, 2026 05:27

@caitaozhan caitaozhan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only some minor comments left!

Comment thread sequence/utils/metrics/__init__.py
Comment thread sequence/utils/metrics/__init__.py
Comment thread sequence/utils/metrics/metric_types.py
Comment thread sequence/utils/metrics/storage.py Fixed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Centralized metrics tracking system

3 participants