Adding `copy_store` convenience method for `Group` #3612

melonora · 2025-12-03T15:51:49Z

[Description of PR]
This PR adds a convenience method for copying the store of a Group to a destination store.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/user-guide/*.md
Changes documented as a new file in changes/
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

codecov · 2025-12-03T16:01:45Z

Codecov Report

❌ Patch coverage is 89.28571% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.97%. Comparing base (65fec71) to head (ae205b5).

Files with missing lines	Patch %	Lines
src/zarr/core/group.py	88.88%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3612      +/-   ##
==========================================
+ Coverage   60.90%   60.97%   +0.07%     
==========================================
  Files          86       86              
  Lines       10174    10199      +25     
==========================================
+ Hits         6196     6219      +23     
- Misses       3978     3980       +2

Files with missing lines	Coverage Δ
src/zarr/testing/strategies.py	`97.83% <100.00%> (ø)`
src/zarr/core/group.py	`70.85% <88.88%> (+0.57%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

melonora · 2025-12-03T16:37:04Z

@d-v-b I know right now this won't work when zarr v2 is used. I can add that functionality as well or we say the convenience function is not supported for zarr v2.

melonora · 2025-12-03T17:53:12Z

added zarr v2 support. If in general the code is approved, I will add documentation.

src/zarr/core/group.py

d-v-b · 2025-12-03T18:04:36Z

src/zarr/core/group.py

+        store: StoreLike,
+        *,
+        overwrite: bool = False,
+        consolidate_metadata: bool | None = None,


why do we need this here? IMO it's simpler if copying is just copying. If people want to add consolidated metadata, they should do that after copying.

adjusted it. It now consolidates if the source store was consolidated, but this does not require an argument. One more thing, I only take into account when consolidation happened at the root level. It could be (although the user should not do this) that consolidation happened at the subgroup level. This could be accounted for by checking if member. metadata.consolidated_metadata when looping through members. WDYT should I add this?

I would copy the original group and its children exactly, including their consolidated metadata.

What would be the purpose of copying over stale consolidated metadata?

it keeps the scope of this function simple -- it just copies the original group exactly. If this tool produces a different group, then it's not really copying.

regarding the from_store method, I would open a separate PR for this.

Consolidated metadata is already in the group metadata document for zarr v3 if "kind" is set to "inline".

👍 but doing something other (more or less) than just copying would require interpreting that which is too far IMO.

FYI @joshmoore the method under discussion here is a method on the Group class, not on stores. maybe this is a point of confusion.

IMO the only consolidated metadata-related parameter we need is whether iterating over the child groups uses their consolidated metadata (if present). See the relevant kwarg in the signature of members.

I added both, only thing remaining is probably adding consolidated_metadata in the async group call.

src/zarr/core/group.py

melonora · 2025-12-05T22:55:36Z

@d-v-b @joshmoore I think I have now addressed all comments and I also added documentation, so I think this would be ready for a final round of review. @d-v-b thanks for your patience.

melonora · 2025-12-10T11:02:42Z

@d-v-b @joshmoore , just bumping in case you have some time to see whether you approve the current state:)

src/zarr/core/group.py

d-v-b · 2025-12-10T11:13:44Z

src/zarr/core/group.py

+        self,
+        store: StoreLike,
+        *,
+        overwrite: bool = False,
+        use_consolidated_for_children: bool = True,


we should add a test that checks the signature of this function against the signature of the async version. otherwise, they have a tendency to drift. Here's an example of such a test:

zarr-python/tests/test_api/test_synchronous.py

Lines 26 to 41 in ee0e69a

def test_docstrings_match(callable_name: str) -> None:

"""

Tests that the docstrings for the sync and async define identical parameters.

"""

callable_a = getattr(synchronous, callable_name)

callable_b = getattr(asynchronous, callable_name)

if callable_a.__doc__ is None:

assert callable_b.__doc__ is None

else:

params_a = NumpyDocString(callable_a.__doc__)["Parameters"]

params_b = NumpyDocString(callable_b.__doc__)["Parameters"]

mismatch = []

for idx, (a, b) in enumerate(zip(params_a, params_b, strict=False)):

if a != b:

mismatch.append((idx, (a, b)))

assert mismatch == []

so I did not do the actual docstrings itself, but I did write a test for all methods for classes and their async counterparts. I think this is generalizable to other classes as well, though currently only Group and AsyncGroup are tested. With this, already some mismatches are detected which for now are skipped but could be addressed in a follow up PR. Doing so would require deprecating some parameters. If you agree I can create an issue for it.

Please see test_class_method_parameters_match

d-v-b · 2025-12-16T19:34:54Z

src/zarr/core/group.py

+        self,
+        store: StoreLike,
+        *,
+        path: str | None = None,


what does it mean when path is None? is it meaningfully different from defaulting to the empty string?

d-v-b · 2025-12-16T19:36:49Z

tests/test_group.py

+    )
+
+
+def get_method_names(cls):


Suggested change

def get_method_names(cls):

def get_method_names(cls) -> str:

d-v-b · 2025-12-16T19:36:56Z

tests/test_group.py

    from zarr.core.common import JSON, ZarrFormat


+def is_deprecated(method):


Suggested change

def is_deprecated(method):

def is_deprecated(method) -> bool:

d-v-b · 2025-12-16T19:37:22Z

tests/test_group.py

+    ]
+
+
+def get_method_signature(cls, method_name: str):


Suggested change

def get_method_signature(cls, method_name: str):

def get_method_signature(cls, method_name: str) -> dict[str, whatever]:

d-v-b · 2025-12-16T19:37:59Z

tests/test_group.py

+    return src, arr_data, subgroup_arr_data, zarr_format, consolidate_metadata
+
+
+def consolidate(src_store, src, consolidate_metadata, zarr_format):


please add type annotations

d-v-b · 2025-12-16T19:38:28Z

tests/test_group.py

+    ],
+    indirect=True,
+)
+def test_copy_to(copy_to_test_data) -> None:


Suggested change

def test_copy_to(copy_to_test_data) -> None:

def test_copy_to(copy_to_test_data: tuple[int, None | tuple[int, ...], bool]) -> None:

d-v-b · 2025-12-16T19:38:55Z

tests/test_group.py

+    ],
+    indirect=True,
+)
+def test_copy_to_with_path(copy_to_test_data) -> None:


Suggested change

def test_copy_to_with_path(copy_to_test_data) -> None:

def test_copy_to_with_path(copy_to_test_data: tuple[int, None | tuple[int, ...], bool]) -> None:

melonora added 2 commits December 3, 2025 16:31

add copy_store convenience method

e3ee33b

add synchronous call

83550a3

github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Dec 3, 2025

change comment, test not async

d473d6e

melonora added 2 commits December 3, 2025 17:37

fix test

bb1405e

support zarr v2

c62543a

d-v-b reviewed Dec 3, 2025

View reviewed changes

src/zarr/core/group.py Outdated Show resolved Hide resolved

d-v-b reviewed Dec 3, 2025

View reviewed changes

src/zarr/core/group.py Outdated Show resolved Hide resolved

d-v-b reviewed Dec 3, 2025

View reviewed changes

melonora added 4 commits December 3, 2025 22:44

remove use of _iter_chunk_regions

cdbc2f7

change method name

e6e10df

remove consolidate_metadata argument

9c42567

consolidate if consolidated

63c652e

d-v-b reviewed Dec 4, 2025

View reviewed changes

src/zarr/core/group.py Outdated Show resolved Hide resolved

melonora added 8 commits December 5, 2025 01:43

add consolidated_metadata argument

d4924f5

add docstring and argument

e83dda5

add support subgroup consolidated metadata

59b18ea

add argument to docstring

b65d257

add example to docs groups

1056b9e

adjust docs

eadb647

partial fix pre-commit

8c3471c

add to changes

2cbb9b9

github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Dec 5, 2025

d-v-b reviewed Dec 10, 2025

View reviewed changes

src/zarr/core/group.py Outdated Show resolved Hide resolved

d-v-b reviewed Dec 10, 2025

View reviewed changes

src/zarr/core/group.py Show resolved Hide resolved

d-v-b reviewed Dec 10, 2025

View reviewed changes

melonora added 4 commits December 10, 2025 12:26

change to call using self

128b924

obtain consolidated metadata from self

fa95e9c

add test for matching parameters

848811f

implement path and add test

3398325

melonora mentioned this pull request Dec 11, 2025

Writing dask arrays to a ZipStore causes a corrupt zip file #3516

Open

Merge branch 'main' into copy_store

ae205b5

d-v-b reviewed Dec 16, 2025

View reviewed changes

tests/test_group.py

)

def get_method_names(cls):

Copy link

Contributor

d-v-b Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

def get_method_names(cls):

def get_method_names(cls) -> str:

d-v-b reviewed Dec 16, 2025

View reviewed changes

	def test_docstrings_match(callable_name: str) -> None:
	"""
	Tests that the docstrings for the sync and async define identical parameters.
	"""
	callable_a = getattr(synchronous, callable_name)
	callable_b = getattr(asynchronous, callable_name)
	if callable_a.__doc__ is None:
	assert callable_b.__doc__ is None
	else:
	params_a = NumpyDocString(callable_a.__doc__)["Parameters"]
	params_b = NumpyDocString(callable_b.__doc__)["Parameters"]
	mismatch = []
	for idx, (a, b) in enumerate(zip(params_a, params_b, strict=False)):
	if a != b:
	mismatch.append((idx, (a, b)))
	assert mismatch == []

		from zarr.core.common import JSON, ZarrFormat


		def is_deprecated(method):

	def is_deprecated(method):
	def is_deprecated(method) -> bool:

	def get_method_signature(cls, method_name: str):
	def get_method_signature(cls, method_name: str) -> dict[str, whatever]:

		return src, arr_data, subgroup_arr_data, zarr_format, consolidate_metadata


		def consolidate(src_store, src, consolidate_metadata, zarr_format):

	def test_copy_to(copy_to_test_data) -> None:
	def test_copy_to(copy_to_test_data: tuple[int, None \| tuple[int, ...], bool]) -> None:

	def test_copy_to_with_path(copy_to_test_data) -> None:
	def test_copy_to_with_path(copy_to_test_data: tuple[int, None \| tuple[int, ...], bool]) -> None:

Uh oh!

Adding copy_store convenience method for Group #3612

Are you sure you want to change the base?

Adding copy_store convenience method for Group #3612

Conversation

melonora commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

melonora commented Dec 3, 2025

Uh oh!

melonora commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

melonora commented Dec 5, 2025

Uh oh!

melonora commented Dec 10, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Adding `copy_store` convenience method for `Group` #3612

Adding `copy_store` convenience method for `Group` #3612

melonora commented Dec 3, 2025 •

edited

Loading

codecov bot commented Dec 3, 2025 •

edited

Loading