Conversation
I don't know if 2 Timelines will ever exist at once, but if so then this might be a problem. For now it seems to be an okay solution
This reverts commit d7ede60.
Code Coverage OverviewLanguages: Python Python / code-coverage/pytestThe overall coverage in the branch remains at 83%, unchanged from the branch. Show a code coverage summary of the most impacted files.
Updated |
caitaozhan
reviewed
Jun 29, 2026
caitaozhan
left a comment
Contributor
There was a problem hiding this comment.
I left some comments. Please take a look.
caitaozhan
reviewed
Jun 30, 2026
caitaozhan
left a comment
Contributor
There was a problem hiding this comment.
Only some minor comments left!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continuation of implementation of #368 and direct continuation of development after #374.
This PR implements a more generalized central metrics tracking system for SeQUeNCe. The module has been split up until multiple modules:
metrics/__init__.py: The front-facing API methods that are used to interact with the metrics module. Includesenable,record,configure,collect_trial_metrics, andaggregate_trial_metrics.event_types.py: Defines methods for creating and handling event types. Event types are what is used by therecordmethod to record a metric.builtins.py: Built-in event types and metrics. Used for abstraction and categorizing the different types of metrics used throughout SeQUeNCe. Also gives an example of how a user could create their own metrics for their own experiments!metrics.py: Mostly defines a Metric abstract class and the types of Metrics that are available. Each metric includescollectmethod that collects the information from those metrics, and this is typically used with thecollect_trial_metricsmethod.registry.py: Metric registry that provides convenient methods for registering and creating new metrics. All Metric types are stored as a list_metricsin this module.storage.py: Mostly unchanged from MVP implementation of centralized metrics tracking system #374, just all storage methods have been moved to their own module, along with the the time provider classes.Metric Superclass
Metrics have been generalized into certain metric types, that change the behavior of certain methods.
Metricis now an abstract superclass of all the other metric types, which are implemented as data classes.Abstract Properties
Metrics requires the abstract properties
event_typesandoutput_keysto be set:event_typesis an immutable set of the EventTypes that the metric reacts to duringrecord.output_keysis an immutable set of the keys that thecollect()method produces in it's output dictionary. Useful for aggregation of data.collect()is discussed in the next section.Collect Method
Metric also requires an abstract method in the form of
collect(). Thecollect()method is to generalize the globalcollect_trial_metrics()method, which now just calls thecollect()method of each metric and puts them all into one list.collect()is implemented differently and outputs different sets of information based on the metric.CounterPairMetriccollects the failures, successes, and success rate of the passed owner. This helps generalize the successes and failures that all the entanglement protocols produce.LastValueMetriccollects the last scalar field value from matching events. This helps generalize values like the throughput of an app.EventFieldListMetriccollects a list of field values matching a certain event type. This helps generalize giant lists of fidelities, like purified fidelities over time.DeliveryTimeMetriccollects the time to deliver N purified pairs relative to reservation start. This helps generalized values like purified delivery of the entanglement purification protocol (RequestApp, etc.).On Record Method
on_record()is an optional method that is called when the metric is recorded. It is only used byCounterPairMetric, where it updates the internal success and failure counters and updates the success rate.Collect Context
CollectContextis a simple data class that lays out a template context to pass to thecollect()method of each metric. It also tells thecollect()method what it can expect when it gets called. It is only used by thecollect_trial_metrics()method. This need some more work to be properly generalized or at least easily extended by the user.Collect Trial Metrics Method
The
collect_trial_metricsmethod is now more generalized for all types of metrics. All it does now is iterate through the list of metrics and calls thecollect()method of each metric. It uses the createdCollectContextobject with the passed arguments, to pass those arguments through to the metricscollect()methods. However, these arguments are currently specific to certain protocols and are not currently generalized.