Add `write_empty_chunks` parameter to `to_icechunk`

[jakenotjay]

Problem

• Suppose you write a timeseries of data, but its possible for the timeseries to have entirely empty chunks - for example in breakpoint analysis (CCDC)
• Writing an empty chunk i.e. a chunk of entirely the fill_value will currently write no changes
• However, you rely on the knowledge of written chunks for some later process e.g. checking for coverage or checking for a complete time series for re-processing
• You've already "processed" this chunk to be empty but cannot verify it to be so via a written empty chunk (as it wasn't written)
• The current solution would be to either
  • A) change the fill value (but now when you pull truly empty chunks it'll show some other value instead of nan)
  • B) pass a sentinel-value e.g. "Infinity" and then remask back to nan

Proposed Solution

• I'd like an additional option for to_icechunk that allows me to set write_empty_chunks to true
• This mirrors the same API as xarrays to_zarr
  • https://docs.xarray.dev/en/latest/generated/xarray.Dataset.to_zarr.html

[dcherian]

I think you can pass this in encoding

[TomNicholas]

The current solution would be to either

Just noting that there is technically a 3rd option: also write some kind of "flag array" that stores information about the processing state of each chunk.

[jakenotjay]

Rapid response @dcherian!

I think you can pass this in encoding

I did look at the zarr specs but couldn't seem to find a reference to it , is there documentation for the supported encoding values?

[jakenotjay]

The current solution would be to either

Just noting that there is technically a 3rd option: also write some kind of "flag array" that stores information about the processing state of each chunk.

I suppose you'd basically create a new band that would be 1 for all written chunks. Doesn't matter then whether or not the value matters, just that its written in the first place. And pull that chunk, at that point though, thats just a workaround for forcing the write of the chunk.

[dcherian]

anything in encoding = {"arrayname": dict[..]} gets passed directly to zarr when writing "arrayname".

[jakenotjay]

anything in encoding = {"arrayname": dict[..]} gets passed directly to zarr when writing "arrayname".

In which case, I don't see anything in the zarr spec for always writing empty chunks, only for setting the fill_value.

https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#fill-value

I've found your commit if this is what you mean in xarray's implementation of this - pydata/xarray@e510a9e

It does look like xarray's to_zarr method in particular would support this encoding value, but given its not in the v3 spec (again correct me if I'm wrong) I'm hesitant to use it, especially if I have to migrate my whole dataset to use a new sentinel band for initialized chunks in the future when something changes in the xarray source. Ideally it would be either part of the spec, or an explicit API parameter (like to_zarr), especially given its undocumented.

If your confident that this behaviour will remain in xarray @dcherian I'm happy to use it, as it solves my problem elegantly.

[dcherian]

The "fill value" section says

Provides an element value to use for uninitialised portions of the Zarr array.

the write_empty_chunks kwarg is a Zarr-python specific kwarg that avoids writing empty chunks. All Zarr implementations should treat an missing chunk binary file as "unitialized" and return a chunk of the right shape filled with fill_value.