feat: backward-compatible Python API improvements (umbrella) by HCookie · Pull Request #102 · ecmwf/qubed

HCookie · 2026-06-22T14:40:41Z

Warning

This is the full set of changes that were needed to restore functionality for Forecast-In-A-Box.
They were agentically built, and have no care for correctness to the vision of qubed.
Let this serve as a showcase as to what was needed, to begin discussion.

Important

This PR has been split into focused, independently-mergeable PRs below.
Each can be reviewed and merged on its own. This PR represents the sum of all changes.

Split PRs

PR	Scope	Status
#104	fix(coordinates): consistent leading-zero handling, safe intersection, type coercion	Draft
#105	fix(merge): safe cross-Qube dimension merging with translation map	Draft
#106	feat(serde): add tree JSON serialisation format	Draft
#107	feat(datacube): deterministic dimension ordering in from_datacube	Draft
#108	feat(py_qubed): ergonomic Python API with immutable ops and new surface	Draft

Each PR is independently mergeable. #108 includes all Rust-level dependencies so it can land on its own if preferred as a single merge.

Full summary of changes

Core Rust library (`qubed/src/`)

Coordinates (coordinates/mod.rs, coordinates/ops.rs) — #104

Fix coordinate intersection to return Empty instead of panicking on cross-type or empty operands
Implement Mixed coordinates to_string() formatting (was todo!())
Fix leading-zero detection: consistent across multi-value coordinate strings
Add type coercion between Strings and Integers when extending coordinates

Merge (merge.rs) — #105

Build a dimension translation map before merging, so two Qubes with dimensions interned in different orders merge correctly
Handle the case where a dimension exists only in other during node_merge

Qube (qube.rs) — #105

Add Clone for Node and Qube
Add axes(), dimensions(), all_dim_ids(), get_or_intern_dim(), prepend() methods
Fix drop_recurse to iteratively splice out consecutive matching dimensions

Datacube (datacube.rs) — #107

from_datacube sorts remaining dimensions alphabetically for deterministic tree structure

Serialisation (serde/json.rs) — #106

Add to_tree_json / from_tree_json: recursive format with key, values, metadata, children fields

Python bindings (`py_qubed/src/lib.rs`) — #108

Breaking changes:

Class renamed from PyQube to Qube on the Python side
drop() and squeeze() return a new Qube instead of mutating in place
to_datacubes() and all_unique_dim_coords() return native Python types
from_arena_json() now accepts str | dict

New API surface:

Qube.empty(), Qube.is_empty()
Qube.from_json() / to_json(), from_tree_json() / to_tree_json()
axes(), dimensions()
__or__ operator, __copy__ / __deepcopy__ support
from_datacube preserves Python dict insertion order
Datacube dicts accept list, int, float values

Tests & Docs

All tests updated, new test files added
docs/src/python/py_qubed.md rewritten

…serialisation - Fix coordinate intersection to return Empty instead of panicking on cross-type or empty operands - Add wrap_ints/wrap_strs helpers to avoid wrapping empty coordinates - Implement Mixed coordinates to_string formatting - Add type coercion between Strings and Integers in coordinate extension - Fix leading-zero detection to be consistent across multi-value coords - Make from_datacube sort remaining dimensions for deterministic output - Build dimension translation map in merge to handle different interning orders between qubes - Handle dimension-only-in-other case during node_merge - Fix drop_recurse to iteratively splice out consecutive matching dims - Add Clone for Node and Qube - Add axes(), dimensions(), all_dim_ids(), get_or_intern_dim() methods - Add to_tree_json/from_tree_json serialisation format - Add comprehensive unit tests for coordinates, datacube, and merge

…e operations - Rename class from PyQube to Qube (Python-side name) - Add from_json/to_json and from_tree_json/to_tree_json bindings - Make from_arena_json accept str or dict (via py_to_json_value) - Preserve Python dict insertion order as default dimension ordering - Accept lists, ints, floats in datacube dicts (not just strings) - Return native types from to_datacubes and all_unique_dim_coords - Make drop() and squeeze() return new Qube instead of mutating - Add __or__ operator for non-mutating merge - Add axes(), dimensions(), is_empty(), empty() methods - Add __copy__/__deepcopy__ support via Clone - Add py_to_json_value recursive converter for Python->serde_json

- Rename all PyQube references to Qube - Update assertions for native type returns (int instead of str) - Add tests for to_json/from_json and to_tree_json/from_tree_json - Add tests for copy/deepcopy, __or__ operator - Add tests for immutable drop() and squeeze() - Add tests for axes(), dimensions(), is_empty() - Add test_qubed_initialisation.py for Qube.empty()

…oject metadata - Remove top-level [project] section from pyproject.toml (now in py_qubed) - Rewrite py_qubed.md for Qube class name, new methods, and native types - Document from_json, to_json, from_tree_json, to_tree_json - Document axes(), dimensions(), is_empty(), __or__, copy support - Update drop/squeeze docs to reflect immutable return semantics

- Keep [project] section in pyproject.toml (main removed version field) - Keep from_py_object and Clone derive on PyQube struct - Keep immutable drop()/squeeze() semantics (return new Qube)

- Add Qube::prepend() on Rust side to wrap tree under new parent node - Add Python prepend() with dict+order signature mirroring append_datacube - Fix pydict_to_datacube to use Coordinates::from_string for slash-splitting - Add from_string_splits_on_slash test - Add prepend tests (wraps tree, order control, multi-value) - Fix __repr__ to return to_ascii() - Update docs to reflect __repr__ behaviour - Remove stale [project] section from root pyproject.toml

HCookie · 2026-06-22T15:24:18Z

+    pub fn from_tree_json(value: Value) -> Result<Qube, String> {
+        let mut qube = Qube::new();
+        let root = qube.root();
+        parse_tree_node(&mut qube, root, &value)?;
+        Ok(qube)
+    }
+}


This is likely unneeded, I can adapt my qube dumps to another format if needed.

I think we might remove this entirely. We can keep it maybe in intermediate versions to have a smoother transition though if it's useful.

We switched the way that the tree is created in Rust with an arena, and changed the standard associated json Qube format at the same time. I think it would be better in the long-term to work with this standard, which we will continue to maintain.

I am using it to walk the qube atm, so a function to do that will be handy

mathleur

Just had a quick go through. Maybe we can separate some of these into different PRs since this is quite a large one and we're addressing different points?

mathleur · 2026-06-23T07:02:19Z

                }
+                Coordinates::Integers(ints) => {
+                    // Try to coerce all new strings to integers
+                    if let Some(converted) = try_strings_to_integers(new_strings) {


Just curious, but why is the conversion to int needed here? I feel this should probably be a separate pre/post-processing function in this case, as it seems likely related to a specific use case? Think this might still be useful in other use cases, and we should implement something, but not sure this is the right place

mathleur · 2026-06-23T07:06:25Z

+    pub fn from_tree_json(value: Value) -> Result<Qube, String> {
+        let mut qube = Qube::new();
+        let root = qube.root();
+        parse_tree_node(&mut qube, root, &value)?;
+        Ok(qube)
+    }
+}


I think we might remove this entirely. We can keep it maybe in intermediate versions to have a smoother transition though if it's useful.

We switched the way that the tree is created in Rust with an arena, and changed the standard associated json Qube format at the same time. I think it would be better in the long-term to work with this standard, which we will continue to maintain.

mathleur · 2026-06-23T07:35:35Z

        // Root should still have "class" children
        let root_node = qube.node(root).unwrap();
        assert!(root_node.children(qube.dimension("class").unwrap()).is_some());



Note to self: did this not work before? Maybe test this functionality more on more complex trees

mathleur · 2026-06-23T07:36:29Z

@@ -1,8 +1,3 @@
-[project]
-name = "qubed"


Is there a reason why we wouldn't want the pyproject project here?

For uv it didn't work sync-ing in this dir, and was very confused

mathleur · 2026-06-23T07:50:35Z

+#### `drop(dims: list[str]) -> Qube`

-Remove one or more dimensions from the tree. Children of removed nodes are re-parented to the grandparent, preserving the rest of the structure. The result is automatically compressed.
+Return a new Qube with one or more dimensions removed. Children of removed nodes are re-parented to the grandparent, preserving the rest of the structure. The result is automatically compressed. The original Qube is not modified.


A lot of these modified functions seem to be so that we do not modify the Qubes in place, is that needed in your application? We had initial design discussions, where we initially decided to modify the Qubes in place for these functions. It might be more consistent with other libraries if we modify in place though and if it doesn't affect the performance too much :)

Sure, I can change my use cases

mathleur · 2026-06-23T08:04:10Z


+    pub fn to_json(&self) -> PyResult<String> {
+        let v = self.inner.to_json();
+        serde_json::to_string(&v).map_err(|e| PyTypeError::new_err(e.to_string()))


That's my bad sorry, but the to_ and from_json were tests, the actual serde format we will be using is to and from_arena_json

Will remove the to and from_json in the Rust version too, so it's less confusing

mathleur · 2026-06-23T08:06:13Z

+    }
+}
+
+fn json_scalar_to_py<'py>(


Note to self: probably should put this in another file/somewhere else if we need them

HCookie · 2026-06-23T08:34:54Z

If you want to see why these changes were made, I've linked the three PR's which update qubed above

HCookie added 6 commits June 22, 2026 14:28

merge: resolve conflicts with origin/main

ca4c688

- Keep [project] section in pyproject.toml (main removed version field) - Keep from_py_object and Clone derive on PyQube struct - Keep immutable drop()/squeeze() semantics (return new Qube)

style: apply cargo fmt formatting

30a4dec

HCookie changed the title ~~fix: backward-compatible Python API improvements and merge correctness~~ fix: more backward-compatible Python API improvements Jun 22, 2026

This was referenced Jun 22, 2026

chore: Refactor qube methods and update tests ecmwf/earthkit-workflows#225

Open

chore: Update qubed version and tests ecmwf/forecast-in-a-box#517

Draft

chore: Update to new qubed ecmwf/earthkit-workflows-anemoi#42

Open

HCookie self-assigned this Jun 22, 2026

HCookie requested a review from mathleur June 22, 2026 15:20

HCookie commented Jun 22, 2026

View reviewed changes

mathleur reviewed Jun 23, 2026

View reviewed changes

HCookie changed the title ~~fix: more backward-compatible Python API improvements~~ feat: backward-compatible Python API improvements (umbrella) Jun 23, 2026

Uh oh!

Conversation

HCookie commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Split PRs

Full summary of changes

Core Rust library (qubed/src/)

Python bindings (py_qubed/src/lib.rs) — #108

Tests & Docs

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mathleur left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HCookie commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HCookie commented Jun 22, 2026 •

edited

Loading

Core Rust library (`qubed/src/`)

Python bindings (`py_qubed/src/lib.rs`) — #108