refactor: extract semantic index crate

[luca-ctx]

Base and provenance

• Base: 5120024f74621bb8932fd28c8ee111d9ee8d5be2 (current origin/main, after merged C1 in PR refactor: extract semantic model boundary #316)
• Approved C2 source: 64e1d7cc41b35d1bad04df6839e94c5f30a879d9
• Approved C2 stable patch ID: fbc6e8c435100df361652e017d2427200a5d162d
• Integration commit: 3de11cea7529d139cee894b9f2bafd203ea5ba1e
• Diff: 64 files, 1,111 insertions, 529 deletions

This ports only C2 onto merged C1. It does not replay the old S1/C1 train.

What changes

Adds ctx-semantic-index as the single production owner of:

• semantic event/document projection and chunk construction;
• generation-pinned Flat-F32 manifests, mmap segments, staging, compaction, recovery, and source reconciliation;
• semantic generation readiness/pinning;
• exact query over caller-supplied, precomputed embeddings.

The CLI keeps composition and policy adapters. It obtains source/query embeddings through the existing model/runtime path and passes vectors into the index. ctx-semantic-model remains the sole owner of model acquisition, runtime loading, E5 identity/projection, and embedding execution.

Behavior preservation

• Persisted root remains data_root/search/semantic.
• Flat format/schema/magic, relative paths, manifest and segment bytes, checksums, staging order, recovery decisions, score bits, tie ordering, generation fingerprints, and resource bounds are unchanged.
• All 34 moved sources existed byte-identically on current main before extraction.
• Nine format/scoring/manifest production authorities remain exact Git-blob moves.
• Semantic test authority is unchanged: 365 #[test] markers before and after (305 remain in CLI, 60 move to the index crate).
• Query embeddings are still generated by the daemon/model path and supplied to SemanticQueryPin::search as &[f32].
• PR Converge JSONL terminal verification and refresh progress #313 typed progress and logical/physical attempt fields are retained in the two overlap files.
• PR test: harden local fixture paths #314 files are untouched; the current raw-output fixture wiring is retained and its test passes uncached.
• There is no duplicate CLI index implementation and no reverse ctx-history-index -> ctx-semantic-index dependency.

CLOC and dependencies

Pinned scc 3.7.0:

Package	Production files	Production CLOC	Limit	Headroom
ctx-semantic-index	28	10,017	20,000	9,983

Surface	Direct dependencies / change
Internal normal	ctx-history-core, ctx-history-index, ctx-semantic-model
External normal	anyhow, fs2, memmap2, rusqlite, serde, serde_json, sha2, thiserror, uuid
Dev only	tempfile
CLI	Adds ctx-semantic-index; removes direct memmap2; test builds use test-support
Forbidden/enforced absent	ctx-history-refresh, CLI reverse imports, model acquisition/loading/embed execution, in-crate SemanticBatchEmbedder implementations

The dependency boundary test checks the exact Bazel closure, reverse paths, frozen model-contract imports, forbidden runtime capabilities, and absence of the old CLI authority.

Cargo, Bazel, release, and SBOM parity

• Registers the Cargo workspace member, Bazel library/test-support/unit targets, LOC inputs, target inventory, format/CI tiers, and dependency boundary.
• Adds the crate to release package audit, configured license materials, dependency materials, and synthetic SBOM coverage.
• All six release-route analysis tests reach the new crate: Linux x64/arm64, macOS x64/arm64, Windows GNU x64, and FreeBSD x64.
• Cargo target metadata resolves on those same six triples with platform-specific runtime dependencies remaining under ctx-semantic-model.
• No external package/version/source/checksum changes were introduced.
• Final lock blobs:
  • Cargo.lock: 25c9eb9fbf8b76ef6de35c020116c60e594de37d
  • MODULE.bazel.lock: 64d40d338f1bbe6db017114730454c7f10c7ef22

Governed validation

All commands ran through ctx-build-governor and the repository wrappers.

ctx-build-governor exec -- scripts/bazelw test //:rustfmt_check --config=test

ctx-build-governor exec -- scripts/bazelw test \
  //crates/ctx-semantic-index:unit_tests \
  //crates/ctx-cli:unit_tests \
  //:semantic_index_dependency_boundary_check \
  //:semantic_model_dependency_boundary_check \
  //:semantic_model_build_cfg_parity_check \
  //:cli_build_cfg_parity_check \
  //:rust_target_inventory_check \
  //:test_tier_inventory_check \
  //:release_sbom_tests \
  //:loc_check \
  //tools/bazel:release_route_analysis_tests \
  --config=test

Focused Bazel result: 17/17 expanded targets passed. The index unit target passed 60/60 tests; CLI units passed 1,331 with 3 ignored.

ctx-build-governor exec -- scripts/cargo-diagnostic.sh test \
  --locked --offline -p ctx-semantic-index --all-features
ctx-build-governor exec -- scripts/cargo-diagnostic.sh check \
  --locked --offline -p ctx-semantic-index --no-default-features
ctx-build-governor exec -- scripts/cargo-diagnostic.sh check \
  --locked --offline --workspace --all-features
ctx-build-governor exec -- scripts/cargo-diagnostic.sh clippy \
  --locked --offline -p ctx-semantic-index --all-targets --all-features -- -D warnings

Cargo all-features passed 60/60 plus doc tests; no-default, workspace all-features, and clippy were clean. Offline dependency trees were also checked for all six release triples.

ctx-build-governor exec -- scripts/bazelw test \
  //crates/ctx-history-index:unit_tests \
  //crates/ctx-cli:raw_output_policy_tests \
  //:package_audit_fast \
  //:release_target_matrix_check \
  //:release_target_matrix_tests \
  //:public_cli_release_targets_tests \
  //:linux_bazel_release_publication_tests \
  //:linux_bazel_controller_receipt_tests \
  //:linux_bazel_release_controller_tests \
  //:stage_github_release_assets_tests \
  //:buildkite_pipeline_check \
  --config=ci --nocache_test_results

Supplemental result: 11/11 passed.

ctx-build-governor exec -- scripts/bazel-affected.sh origin/main
ctx-build-governor exec -- scripts/bazelw test //:ci --config=ci

The affected selector chose //:ci because graph and lock inputs changed, then passed all 123 targets. The separate explicit broad CI run also passed all 123 targets.

An independent read-only adversarial review reported no P0-P3 findings.

Rollback

Revert the single integration commit/PR. This change does not migrate or rename persisted data and does not alter the semantic format, so rollback restores compile-time ownership without a data conversion or store reset.

Residual concerns

• There is no new base-binary-produced Flat-F32 fixture reopened by the candidate. Compatibility evidence rests on exact byte-authority moves plus the complete moved persistence/corruption/recovery/query suites.
• All six release routes and Cargo target graphs were analyzed, but this integration did not perform native execution on every non-host platform. Exact-artifact/native qualification remains release-candidate work.

Draft only; do not merge before independent review.