Manifest preloading might stops early with nested zarr groups

[aladinor]

What happened?

Manifest preloading iteration stops prematurely when a Zarr store has many nested sibling groups. Only the first few groups are processed, leaving the majority of matching arrays without preloaded manifests.

Environment

• icechunk version: 1.1.10
• Python: 3.12
• OS: Linux

Reproduction

A public S3 bucket is available for testing:

import icechunk as ic

# Public bucket with anonymous access
storage = ic.s3_storage(
    bucket='nexrad-arco',
    prefix='KLOT-RT',
    endpoint_url='https://umn1.osn.mghpcc.org',
    anonymous=True,
    force_path_style=True,
    region='us-east-1',
)

# Config to preload coordinate arrays (vcp_time, azimuth, range)
var_condition = ic.ManifestPreloadCondition.name_matches(r'^(vcp_time|azimuth|range)$')
preload_config = ic.ManifestPreloadConfig(
    max_total_refs=1000,  # Very high limit
    preload_if=var_condition,
)
config = ic.RepositoryConfig(manifest=ic.ManifestConfig(preload=preload_config))

repo = ic.Repository.open(storage, config=config)
session = repo.readonly_session('main')

Run with trace logging:

ICECHUNK_LOG=icechunk=debug python script.py

Store Structure

The store contains radar data with the following structure:

/VCP-34/
├── georeferencing_correction/
│   └── vcp_time                       # coordinate array
├── radar_parameters/
│   └── vcp_time                       # coordinate array
├── sweep_0/
│   ├── azimuth                        # coordinate array
│   ├── range                          # coordinate array
│   ├── vcp_time                       # coordinate array
│   └── [data variables...]
├── sweep_1/
│   ├── azimuth, range, vcp_time       # coordinate arrays
│   └── [data variables...]
├── sweep_2/
│   ├── azimuth, range, vcp_time       # coordinate arrays
│   └── ...
... (continues through sweep_9)

Total arrays matching the preload filter: 33

Expected Behavior

All 33 coordinate arrays (vcp_time, azimuth, range) across all groups should be preloaded since:

1. They match the regex filter ^(vcp_time|azimuth|range)$
2. max_total_refs=100
3. Each manifest contains only 1 ref (small 1D coordinate arrays)

Actual Behavior

Only 8 manifests are preloaded, then iteration stops:

Preloading manifest ... for array /VCP-34/georeferencing_correction/vcp_time
Preloading manifest ... for array /VCP-34/radar_parameters/vcp_time
Preloading manifest ... for array /VCP-34/sweep_0/azimuth
Preloading manifest ... for array /VCP-34/sweep_0/range
Preloading manifest ... for array /VCP-34/sweep_0/vcp_time
Preloading manifest ... for array /VCP-34/sweep_1/azimuth
Preloading manifest ... for array /VCP-34/sweep_1/range
Preloading manifest ... for array /VCP-34/sweep_1/vcp_time
[STOPS HERE - no more preloading]

When later accessing arrays in sweep_2 through sweep_9, manifests are fetched on-demand:

# Accessing sweep_5/azimuth triggers manifest download
Downloading manifest ZM0S0EPCPRFWPXZS85SG for /VCP-34/sweep_5/azimuth

# Accessing sweep_9/azimuth triggers manifest download
Downloading manifest FCZW14HH8TR90QMX7KPG for /VCP-34/sweep_9/azimuth

This may impact loading datasets with many nested groups.

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in icechunk.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

[paraseba]

Thanks for reporting this @aladinor !! Can it be that you are hitting the maximum default number of references Icechunk preloads? There is a config variable max_total_refs with a default value of 10_000

[aladinor]

Thanks @paraseba for your prompt reply.

this is the way I am setting repo config

split_config = icechunk.ManifestSplittingConfig.from_dict(
      {
          icechunk.ManifestSplitCondition.AnyArray(): {
              icechunk.ManifestSplitDimCondition.DimensionName("vcp_time"): 12
              * 24
              * 365  
          }
      }
  )

  var_condition = icechunk.ManifestPreloadCondition.name_matches(
      r"^(vcp_time|azimuth|range)$"
  )
  size_condition = icechunk.ManifestPreloadCondition.num_refs(0, 100)

  preload_if = icechunk.ManifestPreloadCondition.and_conditions(
      [var_condition, size_condition]
  )

  preload_config = icechunk.ManifestPreloadConfig(
      max_total_refs=1000,
      preload_if=preload_if,
  )
repo = icechunk.RepositoryConfig(
        manifest=icechunk.ManifestConfig(
            splitting=split_config, preload=preload_config
        ),
    )

I just set max_total_refs=10_000, and rerun the workflow to obtain the same results, only preloading until sweep_1

[paraseba]

You are only preloading manifests that have up to 100 chunk references. Are you should the rest of the manifests match the size and regex conditions?

[aladinor]

Hi @paraseba,

I double checked the regex and increased to num_ref =10_000 and max_total_refs=10_000, and still only preloading a few manifests. The tree contains the same coordinates at all nodes. Thus, not sure why only preload until sweep_1

[aladinor]

Hi @paraseba, I think I found something that might help us to solve this.

in this line https://github.com/earth-mover/icechunk/blob/613477784bbaf7c8b1540ba1320b20cc61a9d897/icechunk/src/repository.rs#L1539C20-L1539C30
where in this iterator

for node in snap
    .iter_arc(&Path::root())
    .filter_ok(|node| node.node_type() == NodeType::Array)
    // TODO: make configurable
    .take(50)  // <-- this guy

goes through arrays in lexicographic order, but the preload_if filter is applied after the .take(50). So for DataTrees with lots of arrays per group (like our radar data), we hit the 50 limit before reaching later sweeps.

Confirmed that changing to .take(100_000) fixes it - all coordinate arrays get preloaded.

Quick options:

1. Make it configurable - add max_arrays_to_scan to ManifestPreloadConfig
2. Remove it - we already have max_total_refs as a budget limit
3. Just bump the default - quickest fix

Let me know what you prefer and I can put together a PR!

[dcherian]

Great find Alfonso!!!

[paraseba]

@aladinor I'm curious about why you want (and are trying hard) to get manifest preloading? Is this an interactive use case?