fix(artifact): snapshot exports consistently and refuse corrupt imports#1019
Merged
Conversation
The FAST export path (the watcher/incremental auto-update) read the raw main-file bytes of a live WAL-mode store: committed rows still in the -wal were missing entirely and mid-checkpoint reads produced torn snapshots (field artifact: four short indexes + btreeInitPage error 11 across ~12 nodes-table pages). Import then installed the torn file byte-for-byte — the integrity check only sanity-checked the projects table. - Export: both quality levels now snapshot via VACUUM INTO (the consistent copy); only the index-stripping remains BEST-only, FAST keeps zstd-3. The fast path pays a full-copy VACUUM INTO now — correctness over speed; a torn team-shared artifact costs far more. - Import: new cbm_store_check_integrity_deep (shallow check + PRAGMA quick_check) — used only on this rare path, hot-path callers keep the cheap shallow check. Reproduce-first: artifact_fast_export_snapshots_live_wal_store exports FAST while the writer connection is open — RED on the old code with 0 of 60 nodes surviving the round-trip (all rows lived in the WAL); GREEN now. store_deep_integrity_detects_page_corruption pins the import guard both directions (healthy passes, zero-banded pages refused). Closes #895 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #895.
Export half:
CBM_ARTIFACT_FAST(the watcher/incremental auto-update path) raw-copied the main file of a live WAL-mode store — committed rows still in the-walwere simply absent, and mid-checkpoint reads were torn (matching the field artifact'squick_checkoutput in the report). Both quality levels now snapshot viaVACUUM INTO; only index-stripping stays BEST-only, FAST keeps zstd-3 — exactly the remedy the report proposed. Perf note: the fast path now pays a full-copy VACUUM INTO per export; correctness first — a torn team-shared artifact costs far more than the copy.Import half: new
cbm_store_check_integrity_deep(shallow +PRAGMA quick_check(1)) used at import only — the three hot-path callers of the shallow check are untouched (quick_check is O(db) and doesn't belong on store-resolution/open paths).Reproduce-first:
artifact_fast_export_snapshots_live_wal_store: export while the writer connection is open → RED on old code with 0 of 60 nodes surviving the round-trip; GREEN now.store_deep_integrity_detects_page_corruption: healthy file passes, zero-banded pages refused — both directions pinned.Full suite 5,991/0, lint-ci clean. Thanks @kriswill — third precise store-integrity report in this series (#896, #897), all three now fixed.