Skip to content

Source of truth in verification results#179

Open
jonasbhend wants to merge 4 commits into
mainfrom
truth-in-verification-results
Open

Source of truth in verification results#179
jonasbhend wants to merge 4 commits into
mainfrom
truth-in-verification-results

Conversation

@jonasbhend

@jonasbhend jonasbhend commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Currently, we don't record the truth used to produce verification results. Therefore, if you recompute evaluation with a different truth (e.g. analysis / stations), the intermediate verification files (currently verif.nc and verif_aggregated.nc) will be overwritten. To avoid this, we propose to make intermediate verification results truth-specific. Also, this may open up the possibility to run evaluation experiments with multiple truth data sources in the future.

Summary of changes

  • introduce truth (label) in rules producing intermediate verification files
  • record truth label and root in NetCDF metadata (convenience)

@jonasbhend jonasbhend requested review from dnerini and frazane June 9, 2026 13:47
@jonasbhend jonasbhend marked this pull request as draft June 9, 2026 15:42
@jonasbhend jonasbhend marked this pull request as ready for review June 10, 2026 06:58
@jonasbhend jonasbhend requested a review from clairemerker June 10, 2026 08:08

@clairemerker clairemerker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good extension, thanks! :) The truth label used will be the one already set in the configs, right?

@dnerini dnerini left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I like this addition, but the approach we used elsewhere is based on hashing the relevant parameters instead of using labels, see also #169

Did you consider doing the same here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants