fix(store): surface SQLITE_CORRUPT from row scans instead of truncating#1018
Merged
Conversation
Every row-scan loop discarded the terminal sqlite3_step code, so SQLITE_CORRUPT mid-scan was indistinguishable from a clean end of results: against a page-corrupted DB, counts answered from covering indexes stayed correct while row fetches died at the first corrupt table page — every query surface returned plausible truncated/empty answers with no error (search total=379 with results=[], Cypher counts diverging from schema counts, BM25 total=0, empty schema sections). All 31 scan loops now capture the terminal code and fail loudly with CBM_STORE_ERR + a store error message when a scan aborts, with per-site cleanup (partial results freed or handed to the caller's normal free path). Two deliberate soft spots, both enrichment-only and documented inline: include_connected neighbor names (no store handle) and the cluster adjacency scan (partial adjacency; primary surfaces already fail loudly). Best-effort schema property-name discovery and two capped helper scans keep their by-design early-exit semantics. Reproduce-first: corrupt_page_scan_returns_error_not_truncation zeroes a 40-page mid-file band per the issue's dd recipe and asserts scans return CBM_STORE_ERR. RED on the old code with the exact reported symptom (OK with 264 of 2000 rows silently truncated); GREEN now. Fixture-health guards make a healthy-scan pass impossible to mistake for a fix. Closes #896 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #896.
Every store row-scan loop had the shape
while (sqlite3_step(stmt) == SQLITE_ROW)with the terminal code discarded —SQLITE_CORRUPTmid-scan was indistinguishable from a clean end of results. Index-covered counts stayed correct while row fetches died at the first corrupt page, producing the report's contradictory error-free output (total: 379, results: [], Cypher vs schema count divergence, BM25total: 0, empty schema/architecture sections).Fix: all 31 scan loops capture the terminal code and fail loudly (
CBM_STORE_ERR+ store error) with per-site cleanup — partial results freed, or assigned to the out-params so the caller's existing free path owns them. Two enrichment-only soft spots are documented inline (include_connected neighbor names, cluster adjacency), and best-effort/capped helpers keep by-design semantics.Reproduce-first (the issue's dd recipe): zero a 40-page mid-file band, scan. RED showed the exact reported bug live —
CBM_STORE_OKwith 264 of 2,000 rows silently truncated; GREEN returnsCBM_STORE_ERR. Two independent scan surfaces asserted; fixture-health guards prevent a vacuous pass (a healthy full scan fails the test as a broken fixture, not as a fixed bug).Full suite 5,990/0, lint-ci clean.
Thanks @kriswill — the field-confusion framing ("symptoms read as extraction bugs") is exactly why this class mattered.