damacy rewrite: prefetcher producer + planner handles#121
Merged
Conversation
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (67.47%) is below the target coverage (70.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## worktree-prefetch #121 +/- ##
=====================================================
- Coverage 58.33% 58.33% -0.01%
=====================================================
Files 56 58 +2
Lines 7962 7881 -81
Branches 1397 1371 -26
=====================================================
- Hits 4645 4597 -48
+ Misses 2747 2724 -23
+ Partials 570 560 -10
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
nclack
force-pushed
the
damacy-rewrite
branch
4 times, most recently
from
80fff84 to
3875ac9
Compare
## Summary - Replace the planner's legacy synchronous zarr metadata/shard cache reads with handles to the prefetch caches. - Delete the old `zarr_meta_cache` / `zarr_shard_cache` path and route planner inputs through `planner_sample`. - Preserve sparse-zarr behavior: missing shard files become fill chunks, while IO/decode/stat errors still fail the sample. - Rename cache tuning/stats to match the new caches: `array_meta`, `shard_index`, and `chunk_layout`. ## Notes - `prefetch_handle` moved to a small standalone header so CUDA-facing planner headers do not pull in `prefetch_cache.h`. - Store-derived validation now happens asynchronously: missing URIs, unsupported source dtypes, per-array rank mismatch, and decode failures surface from `pop()`, not `push()`. - Chunk-layout probing uses an actual ready shard touched by the sample instead of assuming the origin shard exists. - Non-blosc codecs legitimately return no chunk layout; unsupported blosc codecs fail early during prefetch. ## Tests - `cmake --build build --target damacy test_chunk_layout_cache test_prefetcher test_planner` - `env UV_CACHE_DIR=/tmp/uv-cache timeout 90s ctest --test-dir build -R 'test_(planner|chunk_layout_cache|prefetcher)$' --output-on-failure`
nclack
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
damacy.corchestrator into lifecycle, push, plan, pop, and scheduler modules.plan_reserve.zarr_meta_cacheandzarr_shard_cache.array_meta,shard_index, andchunk_layout.Reviewer Notes
src/damacy_plan.c,src/damacy_scheduler.c,src/prefetch/prefetcher.c, andsrc/planner/planner.c.pop(), notpush().admit_seq, so batches preserve push order even when metadata fetches finish out of order.prefetch_handlemoved to a small standalone header so CUDA-facing planner headers do not includeprefetch_cache.h.Tests
env UV_CACHE_DIR=/tmp/uv-cache timeout 90s ctest --test-dir build -R 'test_(planner|chunk_layout_cache|prefetcher)$' --output-on-failuredamacyplus the focused test targets build successfully:cmake --build build --target damacy test_chunk_layout_cache test_prefetcher test_planner