Skip to content

RFC: add member indexing role for one-writer shared snapshots - #275

Draft
hyein-cbio wants to merge 1 commit into
cocoindex-io:mainfrom
hyein-cbio:hyein-cbio/pencilfish
Draft

RFC: add member indexing role for one-writer shared snapshots#275
hyein-cbio wants to merge 1 commit into
cocoindex-io:mainfrom
hyein-cbio:hyein-cbio/pencilfish

Conversation

@hyein-cbio

Copy link
Copy Markdown

Summary

This is an RFC / draft for an experimental one-writer team snapshot workflow.

Some teams do not want every developer machine to incrementally re-index. The intended model is:

  • one leader periodically runs incremental indexing
  • teammates (members) only search a published target_sqlite.db
  • members never write local or shared index state

The new process-local setting is:

COCOINDEX_CODE_INDEXING_ROLE=member   # default: leader

leader remains the default, so existing users are unchanged.

Why this is an RFC

CocoIndex does not recommend sharing live LMDB/SQLite databases over a mounted filesystem. This PR does not do that. Instead it keeps incremental state (cocoindex.db) on the leader and publishes only a completed SQLite snapshot.

Still, this is a new usage model:

  • SMB / mounted-FS atomic replacement is filesystem-dependent
  • members must use the same embedding provider, model, and query params as the leader
  • this is opt-in and still experimental

I would like maintainer feedback on whether this direction is acceptable before treating it as merge-ready.

Behavior

When COCOINDEX_CODE_INDEXING_ROLE=member:

  • MCP startup does not run background indexing
  • MCP search never refreshes or triggers load-time indexing, even if refresh_index=true
  • ccc index, ccc search --refresh, and ccc reset are rejected in that process
  • the published snapshot is opened read-only with a fresh connection per query
  • a matching COCOINDEX_CODE_DB_PATH_MAPPING is required; local index fallback is refused
  • a missing / unreadable snapshot returns an error instead of creating a local index

The role is process-local. Setting it in .mcp.json affects the MCP process only. The bundled SessionStart hook skips ccc index when the parent process already has COCOINDEX_CODE_INDEXING_ROLE=member.

Example member MCP config

{
  "mcpServers": {
    "cocoindex-code": {
      "command": "ccc",
      "args": ["mcp"],
      "env": {
        "COCOINDEX_CODE_INDEXING_ROLE": "member",
        "COCOINDEX_CODE_DB_PATH_MAPPING": "/Users/bob/src/myrepo=/Volumes/team/ccc-indexes/myrepo"
      }
    }
  }
}

Leader workflow: index locally, then sqlite3.Connection.backup() the completed target_sqlite.db and atomically os.replace() it onto the share. Do not publish cocoindex.db.

Non-goals

  • not a live multi-writer shared database
  • not a recommendation to run CocoIndex state on SMB
  • not a project-level setting in .cocoindex_code/settings.yml

Test plan

  • prek run --all-files
  • member MCP search does not call index
  • member ccc index / search --refresh / reset are rejected
  • member search uses a mapped snapshot and does not create local DBs
  • missing snapshot fails without creating files
  • stale same-version daemon without the member-readonly capability is rejected
  • real SMB atomic replace still needs field validation

Feedback especially welcome on:

  1. whether COCOINDEX_CODE_INDEXING_ROLE is the right knob
  2. whether member mode should refuse unmatched DB path mapping (current behavior)
  3. whether snapshot publish should stay documented-only or become a first-class ccc command

Allow a per-process COCOINDEX_CODE_INDEXING_ROLE=member runtime to search
a published target_sqlite.db without starting or refreshing indexes.
Default remains leader so existing users are unchanged.

Member MCP sessions skip background indexing, ignore refresh_index, and
open the mapped snapshot read-only. Explicit ccc index, search --refresh,
and reset are rejected in that process. A matching DB path mapping is
required so members cannot fall back to a local index.

This is an RFC for a one-writer team snapshot workflow. Incremental
state stays on the leader; only a completed SQLite snapshot is published.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant