Option to disable etag/checksum check

[jbusecke]

Just tagging this as it came up during the earthaccess hack day at AGU

Problem

The default etag comparison is an amazing addition to icechunk and seems like a great default, but I would love to have an option to turn the check off for certain scenarios. In particular the MURSST Data is somehow subject to somewhat random file mods. Ultimately this needs to be fixed upstream, but for my workflow it would be nice to confirm that everything else runs smoothly, or just for someone to basically say "I know what I am doing, I want to access the data"

Proposed Solution

Not exactly sure where this would live, perhaps:

repo = ...
session = repo.readonly_session(..., disable_checksum_check=True)

Or maybe it makes more sense to do this on the repo level?

[jbusecke]

cc @jhkennedy

[paraseba]

I see two options:

1. just not setting the checksum when you write, virtual refs are set in Icechunk via Store.set_virtual_ref. In this function the checksum is indeed optional. Of course this is less flexible, nobody can check the cheksums if you do this.
2. Implementing something like what you mention. I'd be in favor of implementing the disable checksum flag to session creation

[TomNicholas]

In particular the MURSST Data is somehow subject to somewhat random file mods.

If this is a dataset-specific problem, then it sounds like the solution should be dataset-specific too, i.e. at write-time. That suggests modifying virtualizarr's use of Store.set_virtual_ref instead of modifying IC's behaviour at read-time. It would be very easy to add an opt-out flag to virtualizarr's .to_icechunk for this.