feat: semantic search with run-grouped embeddings and conversation-unit citations#999
Conversation
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
|
I realized while working on this that because agents tend to monologue while they are working, that it makes sense to concatenate consecutive runs of assistant messages into a single document and embed that instead, which will yield more effective results and better signal-to-noise ratio. Working on this |
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
An OpenAI-compatible embeddings response carrying JSON float arrays is ~4x larger on the wire than the same vectors as encoding_format "base64" (raw little-endian float32 bytes). On a slow link the response transfer dominates each round trip, so the request format directly bounds build throughput. The encoder now asks for base64 and accepts either response shape, so servers that silently ignore the field keep working. A 4xx that names encoding_format downgrades the encoder to plain float requests for its lifetime and redoes the call, so strict servers cost one extra request instead of aborting the build.
…rrency The build loop previously had exactly one embeddings request in flight: kit's Fill walked documents serially, and Batch.Concurrency only parallelizes within one document's chunks (most documents are a single chunk). Against a remote endpoint the build is round-trip-bound, so server-side parallelism could not help. Bump kit to pick up FillOptions.Concurrency (kenn-io/kit#27) and wire a [vector.embeddings] concurrency key (default 4) through EncodeSettings to the fill. Saves into vectors.db remain serialized on one goroutine, preserving the single-writer model. Servers that handle one request at a time queue the extras; concurrency = 1 restores sequential requests.
roborev: Combined Review (
|
SQLite could plan the semantic-search KNN CTE as a co-routine on the inner side of the join, re-running the full brute-force vec0 scan once per chunk-map row: on a 116k-chunk 2560-dim index a single query ran for >9 minutes without completing, far past the daemon's 30s request timeout. With kit's MATERIALIZED + CROSS JOIN fix (kenn-io/kit#27) the same query takes ~315ms.
roborev: Combined Review (
|
Replace the flat [vector.embeddings] endpoint/transport keys with a [vector.embeddings.servers.<name>] table so multiple OpenAI-compatible endpoints can serve the same model. Model identity (model, dimension, max_input_chars, input_suffix) stays global so any server's vectors are interchangeable; per-server config is transport and capacity only (endpoint, api_key_env, timeout, max_retries, batch_size, concurrency). default_server names the server used for search-time query encoding and for builds that don't select one; a single server is the implicit default. embeddings build gains --using <name> to run one build against a different server (e.g. offload to a faster remote box), validated before the build starts on both the CLI and daemon paths. Manager now holds an EncoderSet of named encoders and resolves BuildRequest.Using up front so a mistyped name fails synchronously.
roborev: Combined Review (
|
An unknown BuildRequest.Using name is caller input, so the build route now maps it to 400 with the manager's actionable message via a new vector.ErrUnknownServer sentinel, instead of a generic 500. Regenerate the frontend API client so VectorBuildRequest carries the using field added to the daemon build API.
roborev: Combined Review (
|
The cgo sqlite-vec bindings do not build on Windows, so kit's sqlitevec registers the pure-Go modernc.org/sqlite/vec extension there and expects the modernc "sqlite" driver; opening vectors.db with mattn go-sqlite3 failed every vector test on Windows CI with "no such module: vec0". Split the driver choice and DSN into build-tagged files: mattn sqlite3 with _-prefixed pragma params on Unix+cgo, modernc "sqlite" with _pragma=name(value) params on Windows or without cgo, both applying the same WAL/busy_timeout/synchronous settings and the same file: URI path escaping. The kit smoke test now opens through the same seam. Also make internal/skills TestTargetDir expectations separator-independent with filepath.Join, and upgrade kit to the tagged v0.2.1 release (replacing the branch pseudo-version pin).
Refresh restarted its parking sentinel at 0 on every run, but parking writes are individual autocommit updates: a refresh interrupted between evictSlotOccupant and finalizeEvictions leaves rows parked at negative ordinals, and the next run could park a freshly evicted row in the same session at an already-taken negative ordinal, failing the unique (session_id, ordinal) index deterministically on every retry. Seed the sentinel from the most negative parked ordinal already in the mirror so parked ordinals stay unique across runs; the leftovers themselves self-heal (rescanned rows overwrite their ordinal, full-mode reconciliation deletes vanished ones). Regression test simulates the interrupted run and fails with the UNIQUE violation on the old code.
forceIndexVarLimit hardcoded the mattn raw-connection SetLimit call, so the over-limit chunking tests could never pass on the modernc driver vectors.db now uses on Windows or without cgo. Split the driver-specific call into setConnVarLimit build-tagged files — modernc's sqlite.Limit keeps the limit-forcing guard live there instead of skipping it — and move modernc.org/sqlite to the direct require block now that driver_modernc.go imports it.
roborev: Combined Review (
|
…c-include-cwd-prefixes * origin/main: feat(usage): show session context and token breakdown (kenn-io#982) (kenn-io#989) perf(push): ignore volatile stat fields for session candidacy (kenn-io#1014) feat(settings): support worktree layout mappings (kenn-io#582) (kenn-io#993) fix(config): apply port from config.toml to Config struct (kenn-io#1005) feat(parser): add Qoder session support (kenn-io#1013) fix(usage): wire agent exclusions through usage filters (kenn-io#972) fix(frontend): preserve calendar range picker selections (kenn-io#1016) feat: semantic search with run-grouped embeddings and conversation-unit citations (kenn-io#999) feat(parser): add ZCode SQLite sync support (kenn-io#1003) (kenn-io#1012) fix(sync): drop unchanged opencode-family container sessions (kenn-io#1015) fix(sync): skip local git discovery for foreign-machine sessions (kenn-io#1008) fix(activity): count subagent sessions in activity report cost (kenn-io#1006) feat(i18n): add Korean (ko) locale support (kenn-io#1002)
Adds opt-in semantic/vector search over conversation content alongside the existing substring/regex/FTS modes, and gives every content-search match — in every mode, on all three backends — a conversation-unit citation.
Semantic search
session search --semanticand--hybrid(reciprocal-rank fusion of the vector and FTS legs), plus--scope top|all|subordinateto control whether subordinate evidence (sidechain runs, subagent/fork sessions) is shown; subordinate hits are rank-penalized and annotated, never silently hidden.embeddings build/list/activate/retiremanage the index: through the daemon when one is running, directly (flock-guarded) otherwise.servewires a debounced after-sync scheduler when[vector]is enabled.[vector]in config.toml; Ollama quickstart in the docs). An optionalinput_suffixappends a client-side terminator to every embedded text for models that need one (e.g. Qwen3-Embedding's<|endoftext|>under llama.cpp); it joins the generation fingerprint.session messages --around N --before/--after --roleretrieves context windows around any hit on all three backends;session search --context Ninlines them.Conversation-unit citations
Every match now carries
ordinal_range: [start, end]— the conversation unit enclosing the anchor — plussubordinate,relationship,parent_session_id, andis_sidechain.ordinalremains the exact matched message in every mode.#start-end @anchorwith asubmarker; MCPsearch_contentcarries the same fields; the OpenAPI schema and generated client are updated.SQLite DSN hardening
Read-only connections were silently read-write: mattn/go-sqlite3 ignores
mode=rowithout afile:URI scheme. Fixed for sessions.db, vectors.db, and every foreign-app parser DB (which were also being converted to WAL on read). The fix exposed and fixed a WAL close-ordering bug in the resync swap flow. Paths are percent-escaped; read-only enforcement is pinned by tests.Architecture
Vectors live in a separate
vectors.db(SQLite + sqlite-vec via go.kenn.io/kit), a mirror keyed by resync-stable doc keys with generations fingerprinted by model/dimension/unit-scheme config; a mirror schema version gates cross-version reads (rebuild-required surfaces as 501 with remediation) and resets stale mirrors on writable opens. Staleness, first-build progress, and endpoint outages surface as distinct errors across CLI/HTTP/MCP. Seedocs/semantic-search.md(usage) anddocs/semantic-search-internals.md(unit model, doc keys, derivation invariants, fusion, error taxonomy).Where to look
internal/db/messages.go,internal/db/unit_range.go— run reducer and shared unit-range derivation (the correctness core; reducer-equivalence property test).internal/vector/— mirror, generations, chunk-anchor resolution, build orchestration, encoder, search.internal/db/search_content*.go— semantic/hybrid modes, unit-granularity fusion, scope filtering, lexical citation enrichment.internal/postgres/unit_range.go,internal/duckdb/unit_range.go— SQL-only backend seams over the shared resolvers.cmd/agentsview/embeddings.go,embed_scheduler.go— CLI group, serve wiring, scheduler.internal/vector/encoder.go,internal/vector/build.go— build throughput: requests useencoding_format: "base64"(~4x smaller responses, with transparent float fallback for servers that reject or ignore the field), and a[vector.embeddings] concurrencykey (default 4) embeds documents in parallel via kit's newFillOptions.Concurrency(vector: Fill concurrency and sqlitevec KNN join-order fix kit#27; go.mod pins that PR's commit and should move to a tagged kit release before merge). Saves stay serialized, preserving the single-writer model. Sequential float-JSON requests left builds round-trip-bound against remote endpoints; measured on a slow WireGuard link, these two changes took a full-archive build from ~46 to ~700 chunks/min.Limitations
pg serve/DuckDB validate and report it unavailable (citations work on all three).🤖 Generated with Claude Code