Skip to content

feat: semantic search with run-grouped embeddings and conversation-unit citations#999

Merged
wesm merged 116 commits into
mainfrom
spectacled-peacock
Jul 7, 2026
Merged

feat: semantic search with run-grouped embeddings and conversation-unit citations#999
wesm merged 116 commits into
mainfrom
spectacled-peacock

Conversation

@wesm

@wesm wesm commented Jul 5, 2026

Copy link
Copy Markdown
Member

Adds opt-in semantic/vector search over conversation content alongside the existing substring/regex/FTS modes, and gives every content-search match — in every mode, on all three backends — a conversation-unit citation.

Semantic search

  • session search --semantic and --hybrid (reciprocal-rank fusion of the vector and FTS legs), plus --scope top|all|subordinate to control whether subordinate evidence (sidechain runs, subagent/fork sessions) is shown; subordinate hits are rank-penalized and annotated, never silently hidden.
  • Embedding documents are run-grouped: each user message is one document, and each unbroken run of assistant/tool messages between user turns is concatenated into one document (~25x fewer assistant-side documents than per-message embedding). Semantic hits anchor on the message containing the best-matching chunk's center and carry the run's ordinal span.
  • embeddings build/list/activate/retire manage the index: through the daemon when one is running, directly (flock-guarded) otherwise. serve wires a debounced after-sync scheduler when [vector] is enabled.
  • Embeddings come from any OpenAI-compatible endpoint ([vector] in config.toml; Ollama quickstart in the docs). An optional input_suffix appends a client-side terminator to every embedded text for models that need one (e.g. Qwen3-Embedding's <|endoftext|> under llama.cpp); it joins the generation fingerprint.
  • session messages --around N --before/--after --role retrieves context windows around any hit on all three backends; session search --context N inlines them.

Conversation-unit citations

Every match now carries ordinal_range: [start, end] — the conversation unit enclosing the anchor — plus subordinate, relationship, parent_session_id, and is_sidechain. ordinal remains the exact matched message in every mode.

  • Row cardinality is mode-specific by design: lexical modes stay grep-like (one row per matching source row), semantic returns one row per embedded unit, hybrid one row per unit with exact-match anchors.
  • Lexical and hybrid unit-less rows derive their unit structurally from the messages/sessions tables — deterministic, identical on SQLite/PostgreSQL/DuckDB, and independent of whether a vector index exists. A property test pins derivation to exact equivalence with the embedding reducer's unit spans.
  • Derivation is post-scan and O(page): batched correlated point lookups with run sharing, ~0.4-0.65ms added per 50-hit page on the gated benchmarks.
  • Surfaces: CLI renders #start-end @anchor with a sub marker; MCP search_content carries the same fields; the OpenAPI schema and generated client are updated.

SQLite DSN hardening

Read-only connections were silently read-write: mattn/go-sqlite3 ignores mode=ro without a file: URI scheme. Fixed for sessions.db, vectors.db, and every foreign-app parser DB (which were also being converted to WAL on read). The fix exposed and fixed a WAL close-ordering bug in the resync swap flow. Paths are percent-escaped; read-only enforcement is pinned by tests.

Architecture

Vectors live in a separate vectors.db (SQLite + sqlite-vec via go.kenn.io/kit), a mirror keyed by resync-stable doc keys with generations fingerprinted by model/dimension/unit-scheme config; a mirror schema version gates cross-version reads (rebuild-required surfaces as 501 with remediation) and resets stale mirrors on writable opens. Staleness, first-build progress, and endpoint outages surface as distinct errors across CLI/HTTP/MCP. See docs/semantic-search.md (usage) and docs/semantic-search-internals.md (unit model, doc keys, derivation invariants, fusion, error taxonomy).

Where to look

  • internal/db/messages.go, internal/db/unit_range.go — run reducer and shared unit-range derivation (the correctness core; reducer-equivalence property test).
  • internal/vector/ — mirror, generations, chunk-anchor resolution, build orchestration, encoder, search.
  • internal/db/search_content*.go — semantic/hybrid modes, unit-granularity fusion, scope filtering, lexical citation enrichment.
  • internal/postgres/unit_range.go, internal/duckdb/unit_range.go — SQL-only backend seams over the shared resolvers.
  • cmd/agentsview/embeddings.go, embed_scheduler.go — CLI group, serve wiring, scheduler.
  • internal/vector/encoder.go, internal/vector/build.go — build throughput: requests use encoding_format: "base64" (~4x smaller responses, with transparent float fallback for servers that reject or ignore the field), and a [vector.embeddings] concurrency key (default 4) embeds documents in parallel via kit's new FillOptions.Concurrency (vector: Fill concurrency and sqlitevec KNN join-order fix kit#27; go.mod pins that PR's commit and should move to a tagged kit release before merge). Saves stay serialized, preserving the single-writer model. Sequential float-JSON requests left builds round-trip-bound against remote endpoints; measured on a slow WireGuard link, these two changes took a full-archive build from ~46 to ~700 chunks/min.
  • Frontend diff is regenerated API-client output only; the web UI remains FTS-only in this release.

Limitations

  • Semantic/hybrid search is SQLite-archive only; pg serve/DuckDB validate and report it unavailable (citations work on all three).
  • Metadata filters post-filter the vector leg (over-fetch mitigates recall loss); narrow scopes can under-fill a page past the batched FTS-leg cap.
  • Citation derivation adds ~19-21% to content-search page latency (sub-millisecond absolute); levers (covering index, statement cache) documented but not pulled.
  • Draft: kept open for real-world exercise before merge.

🤖 Generated with Claude Code

@roborev-ci

roborev-ci Bot commented Jul 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (75df6f3)

Summary verdict: Changes need attention before merge due to one high-severity vector build correctness issue and two medium scheduler/config issues.

High

  • Location: internal/vector/encoder.go:75
    Problem: Permanent() treats every 4xx except 429 as a per-document rejection, and skipPermanentEncodeError stamps those documents as skipped. Config/auth failures like 401/403/404 or an invalid model 400 could permanently mark the corpus as embedded-without-vectors and activate the generation, so fixing the endpoint later would not retry without a full rebuild.
    Fix: Only skip errors that are known to be document-specific; abort builds for auth, missing endpoint/model, timeout-like, and other request/config-level 4xx responses.

Medium

  • Location: cmd/agentsview/embed_scheduler.go:183
    Problem: A backstop tick that collides with an already-running build only sets pendingBackstop; it does not arm the debounce timer. If no later sync notification arrives, full reconciliation is deferred until the next backstop interval despite the intended retry behavior.
    Fix: When a backstop tick cannot start, schedule a debounced retry carrying Backstop: true rather than only recording the pending flag.

  • Location: cmd/agentsview/embeddings.go:258
    Problem: embeddings build loads the current vector config, but when a daemon is active it sends only BuildRequest flags to the daemon. The daemon then builds with the model/dimension/endpoint it captured at startup, so a user who edits [vector.embeddings] and runs embeddings build --full-rebuild can silently rebuild the old configuration.
    Fix: Include a config fingerprint/generation in the daemon build request and reject mismatches with a restart/reload instruction, or make the daemon reload/reconstruct the vector manager before building.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 16m55s

@roborev-ci

roborev-ci Bot commented Jul 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (4bc3800)

High-risk issue found: semantic indexing can permanently skip all documents on config/API 4xx failures, plus two medium API-surface/security concerns.

High

  • internal/vector/encoder.go:86: Treating every non-auth 4xx as permanent makes config/request failures like a wrong /embeddings URL or missing model return Permanent() == true. skipPermanentEncodeError can then stamp-skip every document, auto-activate an empty generation, and prevent future incremental builds from retrying those docs.
    • Fix: Only skip-stamp errors known to be document-specific. Abort builds on route/model/schema/config 4xx responses such as 404/405/415, and add a regression test for an endpoint returning 404 for all inputs.

Medium

  • internal/server/huma_routes_embeddings.go:35: Embeddings routes are registered only when an EmbeddingsManager is present, but agentsview openapi builds the spec without one. As a result, /api/v1/embeddings/* is omitted from OpenAPI and the generated TypeScript client has no embeddings service.

    • Fix: Register the routes unconditionally and return 501/503 when the manager is nil, or make OpenAPI generation install a stub manager.
  • internal/server/huma_routes_search.go:34, internal/vector/encoder.go:154: GET /api/v1/search/content accepts mode=semantic/hybrid, which can synchronously embed attacker-controlled pattern values using the user’s configured embeddings endpoint and bearer key. In local-only mode, a malicious website can issue blind cross-origin GETs to spend embeddings quota or trigger authenticated outbound requests.

    • Fix: Treat semantic/hybrid search as credential-using side-effect behavior. Move those modes to a POST endpoint covered by the existing Origin/CSRF guard, or reject semantic/hybrid GETs unless they include a same-origin signal that no-CORS browser requests cannot supply.

Reviewers: 2 done | Synthesis: codex, 13s | Total: 19m3s

@roborev-ci

roborev-ci Bot commented Jul 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (ac09868)

Summary verdict: Medium issues remain; no Critical or High findings were reported.

Medium

  • Location: internal/vector/mirror.go:172
    Problem: Refresh always starts the parking sentinel at 0, so after a canceled or crashed refresh leaves a row parked at a negative ordinal, the next refresh can try to park another row in the same session at the same negative ordinal and hit the (session_id, ordinal) unique index. That can wedge future embedding builds until manual cleanup.
    Fix: Initialize the sentinel below the current minimum negative ordinal, or choose a per-session unused negative ordinal before each park; add a recovery test with a pre-existing negative ordinal.

  • Location: internal/vector/encoder.go:87
    Problem: hasDocumentSpecificEmbeddingError treats any 400/413/422 body containing broad words like token, context, or content as document-specific. A misconfigured endpoint returning 400 invalid token or 400 unsupported content-type would be skip-stamped as permanent for documents, potentially activating an index with missing vectors instead of aborting the build.
    Fix: Match only specific input-size/policy phrases, and add negative tests for auth/config/media-type style 400 responses.

  • Location: frontend/src/lib/api/generated/services/SearchService.ts:74
    Problem: The generated content-search client exposes mode: 'semantic' | 'hybrid' but has no way to send the required X-AgentsView-Search-Intent: semantic header. Calls made through this generated method will be rejected with 403 by humaSearchContent.
    Fix: Regenerate or adjust the client/OpenAPI schema so the header is accepted and sent for semantic/hybrid searches, and cover it with a generated-client test.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 17m15s

@roborev-ci

roborev-ci Bot commented Jul 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (a6daeea)

Medium issue found; no critical or high findings.

Medium

  • internal/vector/build.go:103: Normal builds only force a full mirror refresh for first build, backstop, full rebuild, or include-automated scope changes. After a parser/data-version resync swaps sessions.db, vectors.db can keep its old watermark, so the next scheduled build may scan only sessions with ended_at >= watermark and miss older sessions whose parsed content, ordinals, or deletions changed. This can leave stale vectors and mirror rows until a manual full rebuild or later backstop.
    • Fix: Track archive generation/data-version in vector_meta, or have resync completion force the next embedding build with Backstop: true. Add a regression test for an older-than-watermark session changed by resync.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 19m2s

@wesm

wesm commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

I realized while working on this that because agents tend to monologue while they are working, that it makes sense to concatenate consecutive runs of assistant messages into a single document and embed that instead, which will yield more effective results and better signal-to-noise ratio. Working on this

@roborev-ci

roborev-ci Bot commented Jul 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (f756b01)

Medium-risk issues found: semantic/vector search can underfill results in scoped queries, incremental embedding refresh can miss historical imports, and MCP search context may expose filtered system messages.

Medium

  • internal/mcp/tools.go:501
    search_content context returns adjacent messages through toContextMessages without applying the MCP system-message filter used by get_messages. An MCP client could request context > 0 around a known user/assistant message and receive IsSystem or system-prefixed rows in context_before / context_after.
    Fix: Apply the same filtering policy as get_messages before returning context, and consider passing ExcludeSystem: true for MCP search requests.

  • internal/vector/search.go:92
    The vector query asks for only limit chunk-level hits before rolling them up to document hits. A single multi-chunk message can consume the raw result set, causing a request for N results to return fewer distinct messages.
    Fix: Overfetch chunk hits before RollupByDocument, or query in batches until enough unique documents are available, then truncate to the requested limit.

  • internal/db/search_content.go:709
    Semantic search fetches a fixed global top-K before applying project/agent/date/session filters. This can return empty or underfilled pages when global top hits are out of scope even though in-scope hits exist lower in the vector ranking. The hybrid vector leg has the same issue.
    Fix: Push session-scope metadata into the vector query, or iteratively request deeper vector results until enough filtered hits are found or an exhaustion cap is reached.

  • internal/db/messages.go:467
    Incremental embedding refreshes use sessions.ended_at >= refresh_watermark, so newly synced/imported historical sessions whose ended_at predates the current watermark are skipped until a manual or backstop full refresh.
    Fix: Base incremental scans on a modification/discovery signal such as file_mtime, local_modified_at, or changed session IDs from sync rather than ended_at alone.


Reviewers: 2 done | Synthesis: codex, 12s | Total: 20m14s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (eb0af05)

Summary verdict: medium-risk issues remain around semantic search parity, vector refresh correctness, and MCP system-message filtering.

Medium

  • internal/service/http.go:467
    HTTP-backed semantic/hybrid searches only translate 501 responses to ErrSemanticUnavailable; a server-side ErrSemanticTransient is returned as HTTP 503 and currently becomes an opaque generic error. This breaks direct-vs-daemon error parity and prevents callers from detecting retryable embedding endpoint failures.
    Fix: Preserve the 503 response body in getJSON and map semantic/hybrid /search/content 503s that wrap the transient sentinel back to an error wrapping db.ErrSemanticTransient.

  • internal/vector/mirror.go:352
    Evicted rows are parked at -1, -2, ... with a per-refresh sentinel reset to zero. If a crash or partial refresh leaves a negative parked row behind, the next refresh can try to park another row in the same session at the same negative ordinal and hit the unique (session_id, ordinal) constraint, wedging future builds.
    Fix: Initialize the sentinel below the current minimum ordinal for the affected session/global mirror, or clean up existing negative parked rows before new evictions.

  • cmd/agentsview/main.go:215
    After a startup full resync, the vector scheduler receives only a normal sync notification. If vectors.db already has a refresh watermark, the follow-up build is incremental and scans only sessions with ended_at >= watermark, so parser/data-version resync changes to older sessions can leave stale embeddings until the periodic backstop runs.
    Fix: Force a vector backstop/full mirror refresh after successful runInitialResync, or clear the vector refresh watermark when the archive is rebuilt.

  • internal/mcp/tools.go:496
    The new MCP search_content context path returns ContextBefore and ContextAfter through toContextMessages without applying the MCP system-message suppression used by other MCP transcript tools. An MCP client can call search_content with context > 0 and a query near a system row, exposing stored system prompts or instructions through context_before or context_after.
    Fix: Filter context messages before returning them from MCP, reusing the existing isSystemMessage / role filtering path or filterAndMapMessage, and add a regression test covering system messages in search context windows.


Reviewers: 2 done | Synthesis: codex, 14s | Total: 20m14s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (cd7ab3a)

Medium-risk issues remain before merge.

Medium

  • cmd/agentsview/embed_scheduler.go:183
    A failed or collided backstop tick only sets pendingBackstop; if no later sync notification arrives, the full reconciliation is deferred until the next backstop interval, which is 24h by default.
    Fix: When a backstop tick returns !started or err != nil, arm the debounce timer so the pending backstop is retried soon.

  • internal/server/huma_routes_search.go:154
    SearchContent can return the new FTS-unavailable error for fts and hybrid, but the HTTP handler only maps semantic unavailability to 501; FTS capability failures fall through as HTTP 500.
    Fix: Export or wrap the FTS-unavailable sentinel and map it to a capability response, updating the HTTP client mapping as needed.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 16m54s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (e8fa6e5)

Medium findings require changes before merge.

Medium

  • internal/db/search_content.go:80
    ordinal_start uses omitempty, so a valid semantic/hybrid unit range starting at ordinal 0 serializes without ordinal_start while still emitting ordinal_end, making the API response incomplete for first-message ranges.
    Fix: use pointer fields or custom JSON output so semantic/hybrid range fields are emitted when present even if the start is 0, while lexical matches can still omit them.

  • internal/vector/encoder.go:179
    [vector.embeddings].endpoint is used directly to build the outbound embeddings request, and attemptEncode sends embedding inputs plus Authorization: Bearer <api key> when configured. VectorConfig.Validate only checks that the endpoint is non-empty, so non-loopback http:// endpoints are accepted, exposing session content and API keys to network observers or MITM on hosted/LAN plaintext services.
    Fix: parse the endpoint during config validation and reject plaintext HTTP for non-loopback hosts by default. Keep http://localhost / loopback allowed for local deployments, and require an explicit allow_insecure override for intentional remote plaintext.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 13m38s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (acdfd3a)

Overall verdict: changes need follow-up for vector build lifecycle and blank-input embedding handling before merge.

Medium

  • internal/vector/manager.go:122 - StartBuild detaches builds onto context.Background(), so API-triggered builds can outlive server shutdown and keep using DB handles while they are being closed. Scheduler-triggered builds have the opposite lifecycle issue: they are not registered with the idle tracker, so a long background build can be canceled by daemon idle shutdown mid-build.

    Fix: Tie build execution to a daemon-scoped lifecycle: track API and scheduler builds as active work, cancel them on shutdown, and wait for the manager to finish before closing DB handles.

  • internal/vector/encoder.go:92 - The permanent document-error classifier does not recognize empty or blank input errors, despite the build path intending to skip whitespace-only poison documents. Since embeddable unit scans can produce empty or whitespace-only user/assistant documents, embedding providers that return 400/422 for blank input can abort every build instead of skipping the bad document.

    Fix: Filter blank embeddable units before embedding, or classify empty/blank-input responses as document-specific permanent errors and add regression coverage.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 18m20s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (f54ce5e)

Summary verdict: One medium-severity issue remains; no high or critical findings were reported.

Medium

  • internal/server/huma_routes_embeddings.go:75 - The HTTP embeddings build route forwards vector.BuildRequest directly, so omitted include_automated is treated as false. This ignores [vector].include_automated = true for API/UI-started builds and can change the mirror scope back to excluding automated sessions, causing reconciliation to remove those rows/vectors.

    Fix: Resolve the configured default on the server side. Use a request type with *bool or an explicit set flag so omitted HTTP requests inherit cfg.Vector.IncludeAutomated, while explicit false remains possible.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 13m41s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (5052554)

The PR has one Medium issue to address; no High or Critical findings were reported.

Medium

  • cmd/agentsview/embed_scheduler.go:183 — When a periodic backstop tick collides with another running build, the scheduler only sets pendingBackstop and does not arm any retry. If no later sync Notify() arrives, the missed full reconciliation is deferred until the next backstop interval, 24h by default, leaving vanished or out-of-scope mirror rows and vectors around much longer than intended.

    Suggested fix: On !started in the backstop branch, keep pendingBackstop = true and reset a debounce or retry timer so the backstop is retried once the build slot frees up. Add a scheduler test for a dropped backstop with no subsequent Notify().


Reviewers: 2 done | Synthesis: codex, 7s | Total: 27m3s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (77ac779)

Medium findings block citation parity across non-SQLite backends and MCP consumers.

Medium

  • Location: internal/postgres/search_content.go:299 and internal/duckdb/store.go:906
    Problem: Non-SQLite content search returns ContentMatch values without populating the new always-serialized ordinal_range, so PG/DuckDB responses emit the zero value [0,0] for most matches and lose citation/lineage parity with SQLite.
    Fix: Add equivalent unit-range derivation for PostgreSQL and DuckDB, or at minimum set [ordinal, ordinal] until full derivation is implemented and document any intentional limitation.

  • Location: internal/mcp/tools.go:581
    Problem: The MCP search_content response drops ordinal_range, subordinate, and lineage fields from service.ContentSearchResult, so semantic/hybrid MCP callers cannot cite the matched unit or distinguish subordinate evidence.
    Fix: Extend contentMatch and the mapping to include OrdinalRange, Subordinate, Relationship, ParentSessionID, and Sidechain.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 16m41s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (b6b9c06)

The PR is close, but there are medium-severity API parity gaps to fix before merge.

Medium

  • internal/mcp/tools.go:511: The MCP search_content response drops new citation fields from db.ContentMatch, including ordinal_range and subordinate/lineage fields. HTTP/CLI callers can see the conversation-unit range, but MCP clients cannot reliably fetch or cite the full unit.

    • Fix: Add the new fields to contentMatch, populate them from m.OrdinalRange, m.Subordinate, m.Relationship, m.ParentSessionID, and m.Sidechain, and add an MCP test asserting ordinal_range is present.
  • internal/service/http.go:468: Daemon-backed semantic search loses the retryable db.ErrSemanticTransient sentinel. The server maps query-time embedding endpoint failures to HTTP 503, but the HTTP backend only maps 501 and returns a generic error for 503, unlike the direct backend.

    • Fix: Preserve the parsed 503 error body in getJSON and have SearchContent wrap/map it to db.ErrSemanticTransient for semantic/hybrid content search.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 18m27s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (d4cd9c1)

Medium severity issue found; no Critical or High findings.

Medium

  • internal/service/http.go:466 - HTTP-backed content search drops the new retryable semantic error taxonomy. The server maps db.ErrSemanticTransient to HTTP 503, but httpBackend.SearchContent only maps 501 and otherwise returns a generic HTTP error, so daemon-backed CLI/MCP callers cannot detect retryable embedding endpoint failures with errors.Is.

    Fix: Special-case HTTP 503 from /api/v1/search/content and return an error wrapping db.ErrSemanticTransient, preserving the response message similarly to the existing 501 handling.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 14m15s

@wesm wesm force-pushed the spectacled-peacock branch from 8db5bdc to cc229a5 Compare July 6, 2026 21:31
@wesm wesm changed the title feat: semantic search over user and assistant messages feat: semantic search with run-grouped embeddings and conversation-unit citations Jul 6, 2026
@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (cc229a5)

Summary verdict: one medium-severity issue remains; no high or critical findings were reported.

Medium

  • internal/db/search_content.go:1144 — Hybrid search only classifies FTS syntax errors returned by QueryContext. SQLite FTS5 errors can also surface during row iteration via rows.Err(), which currently returns a raw SQLite error and becomes a 500 instead of the expected 400 SearchInputError.

    Fix: Wrap the rows.Err() path with classifyFTSError(fmt.Errorf("hybrid search fts leg: %w", err)) and add a hybrid-mode malformed FTS query test.


Reviewers: 2 done | Synthesis: codex, 20s | Total: 17m33s

@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (3ed9d18)

Semantic search changes have two Medium issues to address before merge.

Medium

  • internal/vector/mirror.go:169
    Incremental embedding refresh uses the previous refresh_watermark as an ended_at cutoff, so a later sync/import of an older session can be skipped by after-sync builds until a manual or backstop full reconciliation runs.
    Fix: Drive incremental refresh from actual changed session IDs or sync/file modification state, or force reconciliation when sync can introduce older sessions instead of relying only on ended_at.

  • internal/vector/search.go:112
    Search fetches only limit raw vector chunk hits and then calls RollupByDocument; if several top chunks belong to the same document, the rollup can return far fewer unique results while lower-ranked documents are never fetched.
    Fix: Fetch a larger candidate window or loop until enough unique rolled-up documents are collected, then cap to limit.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 14m49s

@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (67c73a0)

Summary verdict: one medium issue remains; no high or critical findings were reported.

Medium

  • internal/vector/build.go:105 - Incremental vector builds can miss historical content rewrites. The mirror refresh only forces a full refresh for first build, scope changes, backstop, or explicit full rebuild, while the incremental path keys off sessions.ended_at. Parser/data-version resyncs that rewrite older session content can leave stale mirrored text, doc keys, snippets, and embeddings until a future backstop/full rebuild.
    • Suggested fix: track a content/source modification signal for mirror refreshes, or force a backstop/full refresh after full resync/data-version resync so changed historical sessions are re-mirrored.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 16m10s

wesm added 2 commits July 6, 2026 23:49
An OpenAI-compatible embeddings response carrying JSON float arrays is
~4x larger on the wire than the same vectors as encoding_format
"base64" (raw little-endian float32 bytes). On a slow link the
response transfer dominates each round trip, so the request format
directly bounds build throughput.

The encoder now asks for base64 and accepts either response shape, so
servers that silently ignore the field keep working. A 4xx that names
encoding_format downgrades the encoder to plain float requests for its
lifetime and redoes the call, so strict servers cost one extra request
instead of aborting the build.
…rrency

The build loop previously had exactly one embeddings request in flight:
kit's Fill walked documents serially, and Batch.Concurrency only
parallelizes within one document's chunks (most documents are a single
chunk). Against a remote endpoint the build is round-trip-bound, so
server-side parallelism could not help.

Bump kit to pick up FillOptions.Concurrency (kenn-io/kit#27) and wire a
[vector.embeddings] concurrency key (default 4) through EncodeSettings
to the fill. Saves into vectors.db remain serialized on one goroutine,
preserving the single-writer model. Servers that handle one request at
a time queue the extras; concurrency = 1 restores sequential requests.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (a24be57)

Medium severity findings:

  • cmd/agentsview/embed_scheduler.go:193 - Backstop retries can be delayed up to the next backstop interval, defaulting to 24 hours. When a tick fires while another build is running, or a build starts but fails, pendingBackstop is set but no debounce timer is armed. Set pendingBackstop = true and reset the debounce timer when !started || err != nil.

  • internal/vector/encoder.go:289 - Embedding endpoints accept non-loopback plaintext http:// URLs while sending raw session chunks/search text and optional bearer auth. This creates a secret-bearing egress path where LAN or hosted HTTP services expose archive content and API keys to passive network attackers or MITM. Validate endpoints before use: require HTTPS for non-loopback hosts, allow HTTP only for loopback/local servers, or require an explicit insecure opt-in.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 18m21s

SQLite could plan the semantic-search KNN CTE as a co-routine on the
inner side of the join, re-running the full brute-force vec0 scan once
per chunk-map row: on a 116k-chunk 2560-dim index a single query ran
for >9 minutes without completing, far past the daemon's 30s request
timeout. With kit's MATERIALIZED + CROSS JOIN fix (kenn-io/kit#27) the
same query takes ~315ms.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (2ee1fbe)

Summary verdict: One medium issue remains; no high or critical findings were reported.

Medium

  • internal/service/http.go:468 - Daemon-backed semantic/hybrid searches lose the new retryable db.ErrSemanticTransient signal. The server maps query-embedding failures to HTTP 503, but the HTTP service client only remaps 501 responses, so callers receive a generic GET ... HTTP 503 error instead of an error matching the transient sentinel.
    • Fix: Teach the HTTP client to preserve 503 search/content error bodies and wrap db.ErrSemanticTransient, with a test covering daemon-backed semantic transient failures.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 28m22s

wesm added 2 commits July 7, 2026 09:36
Replace the flat [vector.embeddings] endpoint/transport keys with a
[vector.embeddings.servers.<name>] table so multiple OpenAI-compatible
endpoints can serve the same model. Model identity (model, dimension,
max_input_chars, input_suffix) stays global so any server's vectors are
interchangeable; per-server config is transport and capacity only
(endpoint, api_key_env, timeout, max_retries, batch_size, concurrency).

default_server names the server used for search-time query encoding and
for builds that don't select one; a single server is the implicit
default. embeddings build gains --using <name> to run one build against
a different server (e.g. offload to a faster remote box), validated
before the build starts on both the CLI and daemon paths.

Manager now holds an EncoderSet of named encoders and resolves
BuildRequest.Using up front so a mistyped name fails synchronously.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (a8fba36)

Summary verdict: one high-risk correctness issue and one medium API-client gap remain.

High

  • internal/vector/build.go:204
    --full-rebuild of the currently active generation deletes its vectors/stamps while leaving it active. If kitvec.Fill then fails or is canceled, semantic search continues querying the now-partial active generation and silently returns incomplete results.
    Fix: Rebuild active generations through a non-active replacement generation and swap only after full coverage, or mark the active generation unavailable until refill succeeds. Add a regression test for active full-rebuild failure.

Medium

  • frontend/src/lib/api/generated/models/VectorBuildRequest.ts:5
    The generated API model is missing the new using request field that vector.BuildRequest accepts, so TypeScript clients cannot type-safely select a named embeddings server through /api/v1/embeddings/build.
    Fix: Ensure the OpenAPI schema includes using and regenerate the frontend API client.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 16m14s

An unknown BuildRequest.Using name is caller input, so the build route
now maps it to 400 with the manager's actionable message via a new
vector.ErrUnknownServer sentinel, instead of a generic 500.

Regenerate the frontend API client so VectorBuildRequest carries the
using field added to the daemon build API.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (7d46369)

Medium: one issue should be addressed before merge.

  • Medium: internal/vector/mirror.go:182
    Refresh restarts the parking sentinel at 0 on every run, but parked negative ordinals are written non-transactionally. If a refresh is canceled or errors after evictSlotOccupant parks a row and before finalizeEvictions removes/reinserts it, the next refresh can try to park another row in the same session at the same negative ordinal, violating the unique (session_id, ordinal) index and wedging vector refreshes.
    Fix: Before assigning parking ordinals, either clean/recover existing negative parked rows or seed the sentinel below the current minimum negative ordinal for that session/global mirror; preferably make the eviction/upsert/finalize sequence recoverable across interrupted runs.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 23m4s

@wesm wesm marked this pull request as ready for review July 7, 2026 15:21
wesm added 3 commits July 7, 2026 11:28
The cgo sqlite-vec bindings do not build on Windows, so kit's sqlitevec
registers the pure-Go modernc.org/sqlite/vec extension there and expects
the modernc "sqlite" driver; opening vectors.db with mattn go-sqlite3
failed every vector test on Windows CI with "no such module: vec0".
Split the driver choice and DSN into build-tagged files: mattn sqlite3
with _-prefixed pragma params on Unix+cgo, modernc "sqlite" with
_pragma=name(value) params on Windows or without cgo, both applying the
same WAL/busy_timeout/synchronous settings and the same file: URI path
escaping. The kit smoke test now opens through the same seam.

Also make internal/skills TestTargetDir expectations
separator-independent with filepath.Join, and upgrade kit to the tagged
v0.2.1 release (replacing the branch pseudo-version pin).
Refresh restarted its parking sentinel at 0 on every run, but parking
writes are individual autocommit updates: a refresh interrupted between
evictSlotOccupant and finalizeEvictions leaves rows parked at negative
ordinals, and the next run could park a freshly evicted row in the same
session at an already-taken negative ordinal, failing the unique
(session_id, ordinal) index deterministically on every retry.

Seed the sentinel from the most negative parked ordinal already in the
mirror so parked ordinals stay unique across runs; the leftovers
themselves self-heal (rescanned rows overwrite their ordinal, full-mode
reconciliation deletes vanished ones). Regression test simulates the
interrupted run and fails with the UNIQUE violation on the old code.
forceIndexVarLimit hardcoded the mattn raw-connection SetLimit call, so
the over-limit chunking tests could never pass on the modernc driver
vectors.db now uses on Windows or without cgo. Split the driver-specific
call into setConnVarLimit build-tagged files — modernc's sqlite.Limit
keeps the limit-forcing guard live there instead of skipping it — and
move modernc.org/sqlite to the direct require block now that
driver_modernc.go imports it.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (0129a34)

Reviewed the semantic search/vector indexing, embeddings API/CLI, MCP search changes, skills install path, and backend query construction for trust-boundary regressions. I did not find a concrete exploit path introduced by the diff.

No issues found.


Reviewers: 2 total (1 done, 1 skipped) | Synthesis: codex | Total: 39m5s

@wesm wesm merged commit 6c25213 into main Jul 7, 2026
24 checks passed
@wesm wesm deleted the spectacled-peacock branch July 7, 2026 16:45
wesm added a commit to RobSchilderr/agentsview that referenced this pull request Jul 7, 2026
…c-include-cwd-prefixes

* origin/main:
  feat(usage): show session context and token breakdown (kenn-io#982) (kenn-io#989)
  perf(push): ignore volatile stat fields for session candidacy (kenn-io#1014)
  feat(settings): support worktree layout mappings (kenn-io#582) (kenn-io#993)
  fix(config): apply port from config.toml to Config struct (kenn-io#1005)
  feat(parser): add Qoder session support (kenn-io#1013)
  fix(usage): wire agent exclusions through usage filters (kenn-io#972)
  fix(frontend): preserve calendar range picker selections (kenn-io#1016)
  feat: semantic search with run-grouped embeddings and conversation-unit citations (kenn-io#999)
  feat(parser): add ZCode SQLite sync support (kenn-io#1003) (kenn-io#1012)
  fix(sync): drop unchanged opencode-family container sessions (kenn-io#1015)
  fix(sync): skip local git discovery for foreign-machine sessions (kenn-io#1008)
  fix(activity): count subagent sessions in activity report cost (kenn-io#1006)
  feat(i18n): add Korean (ko) locale support (kenn-io#1002)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant