feat(capture): emit synthetic events for oversized SQLite rows - #89
Closed
robbe1912 wants to merge 3 commits into
Closed
feat(capture): emit synthetic events for oversized SQLite rows#89robbe1912 wants to merge 3 commits into
robbe1912 wants to merge 3 commits into
Conversation
Providers like the locally-installed Z.ai Kilo client can carry single
messages > 100 MB (e.g. a multi-file diff summary) that legitimately
exceed ctx's per-value import cap. Previously one oversized row aborted
the entire source import, which in turn made ctx search --refresh
strict refuse to refresh the index for any provider sharing the
invocation.
Skip oversized rows instead:
* Pre-scan oversized row ids on a separate unbounded read connection
(the bounded connection enforces SQLITE_LIMIT_LENGTH even when
computing length(), so the cap must be off to enumerate offenders).
* Exclude those ids at the SQL layer via where id not in (...) —
sqlite3_step() materializes every column of the current row, so
filtering after
ows.next() is too late.
* Keep a per-row try/catch on the data column as a safety net for
edge cases the pre-scan might miss (e.g. multi-byte UTF-8 inserted
between the two connections).
* Surface the skip count via the import summary's failure list so
ctx import and ctx search --refresh both report what was dropped.
The pre-scan runs after schema validation so 'missing required column'
errors for future/incompatible provider schemas still surface first.
This was referenced Jul 7, 2026
Match repo style (ctx-history-capture crate has near-zero inline comments across all provider adapters). Move design rationale to PR description; keep only load-bearing items in code. Also correct provider attribution in the one retained comment: Kilo Code is the VSCode extension, not a z.ai product. z.ai is the coding-plan/model provider; the local sqlite store ctx reads belongs to the Kilo Code extension itself.
Oversized rows (data column > MAX_PROVIDER_SQLITE_VALUE_BYTES) are re-imported as synthetic events carrying a truncated preview plus a _ctx_oversized marker (original_bytes, source_table, source_format, preview_chars, preview_truncated). The marker surfaces as a top-level field on event.payload alongside body so consumers can detect truncation without reaching into body (which is itself capped by provider_capped_json). Adds truncated_oversized_message_rows helper that opens an unbounded read-only connection (the bounded connection refuses to materialize the value) and emits one synthetic row per oversized entry. All three row fetchers (session_message / session_entry / message) extend their result with synthetic rows before returning. Drops the Phase 1 summary.skipped increment in normalize_opencode_sqlite since oversized rows are now counted as imported_events. Regression test verifies summary.failed==0, imported_events>=1, and that exactly one event carries the _ctx_oversized marker with the expected original_bytes / source_table / source_format / message_id.
robbe1912
force-pushed
the
fix/search-refresh-truncated-marker
branch
from
July 7, 2026 13:15
1be13e8 to
7ffd719
Compare
robbe1912
marked this pull request as ready for review
July 7, 2026 13:35
Contributor
|
Thanks for the PR! I am going to resolve the oversized row issue on top of your other PR (#88) and close this one as we are going to do a different approach than synthetic events because those could give the impression that the synthetic data is real data. We would like to either mark the row as skipped for oversize, or somehow fix the oversize issue. Hopefully this makes sense and stay tuned on #88 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on #88. Phase 1 made oversized SQLite rows not abort the source import by skipping them at the SQL layer. The skipped rows themselves were invisible — no event landed in the store, no search hit referenced them,
ctx show sessionsilently omitted them. For providers like the Kilo Code VSCode extension that occasionally carry a single multi-hundred-MB message (e.g. a giant multi-file diff captured by the extension), that meant an entire legitimate session could be silently dropped from history.This change re-imports each oversized row as a synthetic event carrying a truncated preview of the original payload plus a
_ctx_oversizedmarker that records the original byte count and source table. The session stays searchable end-to-end, and downstream consumers can detect truncation explicitly.truncated_oversized_message_rowsthat opens an unbounded read-only connection (the bounded connection refuses to materialize the value, so truncation must run against an uncapped connection — same reasoning asoversized_message_row_idsin fix(capture): skip oversized SQLite rows instead of aborting source #88) and emits oneOpenCodeMessageRowper oversized row, withdataset to a synthetic JSON payload containing the truncated preview (PROVIDER_MAX_TEXT_CHARSchars) and a_ctx_oversizedmarker.opencode_session_message_rows,opencode_session_entry_rows,opencode_message_rows) to append the synthetic rows before returning so they flow through the existing per-row import path unchanged.summary.skippedincrement innormalize_opencode_sqlite: oversized rows are no longer skipped, they are imported as synthetic events and counted asimported_events._ctx_oversizedmarker inopencode_event.payloadas a top-level field alongsidebodyfor downstream visibility. The marker is also nested insidebodybutprovider_capped_jsonwould otherwise lose it for any oversized row whose synthetic preview also exceeds the body cap, so the top-level field is the canonical detection point.Marker shape (at
event.payload.body.ctx_oversized):{ "original_bytes": <byte count of the oversized row>, "preview_chars": 16000, "source_table": "session_message", "source_format": "opencode_sqlite", "preview_truncated": false }Affected scope: only the OpenCode-family SQLite adapter (covers the Kilo Code extension via
KiloSqliteAdapter). JSONL adapters are unaffected.Validation
cargo check -p ctx-history-capture— cleancargo build --workspace— clean, only the 2 pre-existing warnings on thectxbin (ApplyResult::Appliedunused,ProcessState::Running/NotRunningnever constructed) that exist onmaincargo test -p ctx-history-capture --lib tests::native_sqlite— 29 passed, 0 failednative_opencode_truncates_oversized_sqlite_text_value_and_preserves_other_rowsasserts:summary.failed == 0(oversized rows must not be treated as failures)summary.imported_events >= 1(other rows still import)payload.body.ctx_oversizedpayload.body.ctx_oversized.original_bytesmatches the oversized row sizepayload.body.ctx_oversized.source_table/source_formatare setpayload.body.message_idpreserves the original provider row idEnd-to-end verification
Built ctx from this branch and imported a synthetic
opencode.dbfixture (smoke-test schema) containing 2 normal rows + 1 oversized row (MAX_PROVIDER_SQLITE_VALUE_BYTES + 1bytes undertext):All three rows imported cleanly (Phase 1 effect: no
SQLITE_TOOBIGabort), and exactly one synthetic event landed in the store with the_ctx_oversizedmarker atpayload_json.body.ctx_oversized(Phase 2 effect).Depends on
fix/search-refresh-oversized-skipand should land after it.Related
text/content/messagekeys, blocks all kilo imports regardless of oversized handling).