Skip to content

feat(capture): emit synthetic events for oversized SQLite rows - #89

Closed
robbe1912 wants to merge 3 commits into
ctxrs:mainfrom
robbe1912:fix/search-refresh-truncated-marker
Closed

feat(capture): emit synthetic events for oversized SQLite rows#89
robbe1912 wants to merge 3 commits into
ctxrs:mainfrom
robbe1912:fix/search-refresh-truncated-marker

Conversation

@robbe1912

@robbe1912 robbe1912 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Builds on #88. Phase 1 made oversized SQLite rows not abort the source import by skipping them at the SQL layer. The skipped rows themselves were invisible — no event landed in the store, no search hit referenced them, ctx show session silently omitted them. For providers like the Kilo Code VSCode extension that occasionally carry a single multi-hundred-MB message (e.g. a giant multi-file diff captured by the extension), that meant an entire legitimate session could be silently dropped from history.

This change re-imports each oversized row as a synthetic event carrying a truncated preview of the original payload plus a _ctx_oversized marker that records the original byte count and source table. The session stays searchable end-to-end, and downstream consumers can detect truncation explicitly.

  • Add truncated_oversized_message_rows that opens an unbounded read-only connection (the bounded connection refuses to materialize the value, so truncation must run against an uncapped connection — same reasoning as oversized_message_row_ids in fix(capture): skip oversized SQLite rows instead of aborting source #88) and emits one OpenCodeMessageRow per oversized row, with data set to a synthetic JSON payload containing the truncated preview (PROVIDER_MAX_TEXT_CHARS chars) and a _ctx_oversized marker.
  • Extend all three row fetchers (opencode_session_message_rows, opencode_session_entry_rows, opencode_message_rows) to append the synthetic rows before returning so they flow through the existing per-row import path unchanged.
  • Drop the Phase 1 summary.skipped increment in normalize_opencode_sqlite: oversized rows are no longer skipped, they are imported as synthetic events and counted as imported_events.
  • Surface the _ctx_oversized marker in opencode_event.payload as a top-level field alongside body for downstream visibility. The marker is also nested inside body but provider_capped_json would otherwise lose it for any oversized row whose synthetic preview also exceeds the body cap, so the top-level field is the canonical detection point.

Marker shape (at event.payload.body.ctx_oversized):

{
  "original_bytes": <byte count of the oversized row>,
  "preview_chars": 16000,
  "source_table": "session_message",
  "source_format": "opencode_sqlite",
  "preview_truncated": false
}

Affected scope: only the OpenCode-family SQLite adapter (covers the Kilo Code extension via KiloSqliteAdapter). JSONL adapters are unaffected.

Validation

  • cargo check -p ctx-history-capture — clean
  • cargo build --workspace — clean, only the 2 pre-existing warnings on the ctx bin (ApplyResult::Applied unused, ProcessState::Running/NotRunning never constructed) that exist on main
  • cargo test -p ctx-history-capture --lib tests::native_sqlite — 29 passed, 0 failed
  • Regression test native_opencode_truncates_oversized_sqlite_text_value_and_preserves_other_rows asserts:
    • summary.failed == 0 (oversized rows must not be treated as failures)
    • summary.imported_events >= 1 (other rows still import)
    • exactly one event in the session carries a non-null payload.body.ctx_oversized
    • payload.body.ctx_oversized.original_bytes matches the oversized row size
    • payload.body.ctx_oversized.source_table / source_format are set
    • payload.body.message_id preserves the original provider row id

End-to-end verification

Built ctx from this branch and imported a synthetic opencode.db fixture (smoke-test schema) containing 2 normal rows + 1 oversized row (MAX_PROVIDER_SQLITE_VALUE_BYTES + 1 bytes under text):

[ctx] imported: events=3, sessions=1, failed=0, skipped=0
[store] exactly one synthetic oversized event with _ctx_oversized marker
[store]   message_id preserved
[store]   original_bytes matches oversized row size
[store]   source_table=session_message, source_format=opencode_sqlite
[PASS] PR #88 + #89 verified end-to-end through ctx CLI

All three rows imported cleanly (Phase 1 effect: no SQLITE_TOOBIG abort), and exactly one synthetic event landed in the store with the _ctx_oversized marker at payload_json.body.ctx_oversized (Phase 2 effect).

Depends on

Related

Providers like the locally-installed Z.ai Kilo client can carry single
messages > 100 MB (e.g. a multi-file diff summary) that legitimately
exceed ctx's per-value import cap. Previously one oversized row aborted
the entire source import, which in turn made ctx search --refresh
strict refuse to refresh the index for any provider sharing the
invocation.

Skip oversized rows instead:

  * Pre-scan oversized row ids on a separate unbounded read connection
    (the bounded connection enforces SQLITE_LIMIT_LENGTH even when
    computing length(), so the cap must be off to enumerate offenders).
  * Exclude those ids at the SQL layer via where id not in (...) —
    sqlite3_step() materializes every column of the current row, so
    filtering after 
ows.next() is too late.
  * Keep a per-row try/catch on the data column as a safety net for
    edge cases the pre-scan might miss (e.g. multi-byte UTF-8 inserted
    between the two connections).
  * Surface the skip count via the import summary's failure list so
    ctx import and ctx search --refresh both report what was dropped.

The pre-scan runs after schema validation so 'missing required column'
errors for future/incompatible provider schemas still surface first.
robbe1912 added 2 commits July 7, 2026 15:06
Match repo style (ctx-history-capture crate has near-zero inline
comments across all provider adapters). Move design rationale to PR
description; keep only load-bearing items in code.

Also correct provider attribution in the one retained comment:
Kilo Code is the VSCode extension, not a z.ai product. z.ai is the
coding-plan/model provider; the local sqlite store ctx reads belongs
to the Kilo Code extension itself.
Oversized rows (data column > MAX_PROVIDER_SQLITE_VALUE_BYTES) are
re-imported as synthetic events carrying a truncated preview plus a
_ctx_oversized marker (original_bytes, source_table, source_format,
preview_chars, preview_truncated). The marker surfaces as a top-level
field on event.payload alongside body so consumers can detect
truncation without reaching into body (which is itself capped by
provider_capped_json).

Adds truncated_oversized_message_rows helper that opens an unbounded
read-only connection (the bounded connection refuses to materialize
the value) and emits one synthetic row per oversized entry. All three
row fetchers (session_message / session_entry / message) extend their
result with synthetic rows before returning. Drops the Phase 1
summary.skipped increment in normalize_opencode_sqlite since oversized
rows are now counted as imported_events.

Regression test verifies summary.failed==0, imported_events>=1, and
that exactly one event carries the _ctx_oversized marker with the
expected original_bytes / source_table / source_format / message_id.
@robbe1912
robbe1912 force-pushed the fix/search-refresh-truncated-marker branch from 1be13e8 to 7ffd719 Compare July 7, 2026 13:15
@robbe1912
robbe1912 marked this pull request as ready for review July 7, 2026 13:35
@luca-ctx

luca-ctx commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for the PR! I am going to resolve the oversized row issue on top of your other PR (#88) and close this one as we are going to do a different approach than synthetic events because those could give the impression that the synthetic data is real data. We would like to either mark the row as skipped for oversize, or somehow fix the oversize issue. Hopefully this makes sense and stay tuned on #88

@luca-ctx luca-ctx closed this Jul 7, 2026
@robbe1912
robbe1912 deleted the fix/search-refresh-truncated-marker branch July 7, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants