feat(index): crash/hang supervisor — quarantine the culprit file, index the rest#827
Merged
Conversation
…cation
New src/foundation/subprocess.{h,c}: spawn a child, tail its output, and
classify how it ended — clean / exit-nonzero / crash / hang / killed — from
POSIX WIFSIGNALED/WTERMSIG and the Windows NTSTATUS exception exit codes
(0xC0000005 access violation, 0xC00000FD stack overflow). Adds an EINTR-safe
reap loop, a quiet-timeout that kills and reports a hung child (no new output
within the window), and partial-line-safe log tailing.
Generalized from the ad-hoc index spawn in src/ui/http_server.c so the
crash/hang supervisor can reuse one primitive across platforms. The exit-code
to outcome mapping lives in a pure cbm_proc_classify() so the Windows
crash-code path is unit-tested on every platform, not just an untested
Windows branch.
tests/test_subprocess.c: 13 tests — the classifier on all platforms plus real
POSIX spawn/reap of clean, non-zero, SIGSEGV-crash and hang children. Green
under ASan/UBSan.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A single pathological file that hard-crashes the native indexer (SIGSEGV / stack overflow / abort / tree-sitter assertion) previously took down the whole MCP server or CLI (issue #668). index_repository now runs in a worker subprocess: the parent reaps it, and a crash is contained — the parent survives and reports it instead of dying. - src/mcp/index_supervisor.{h,c}: worker-role state + cbm_index_spawn_worker(), which spawns `<self> cli --index-worker index_repository <json> --response-out <tmp>` via the subprocess primitive, reaps, and classifies. fork+exec only; recursion is prevented by the --index-worker argv flag, never an ambient env var (which could misfire and silently run unprotected in-process). - src/mcp/mcp.c: a supervisor gate at the top of handle_index_repository wraps the run unless this process is the worker or CBM_INDEX_SUPERVISOR=0 (kill switch). Clean exit passes the worker's response through; a crash/hang returns a contained-failure response; a spawn failure degrades to the in-process path. - src/main.c: parse --index-worker / --response-out; the worker writes its result to the response file for the parent to read back. - src/ui/http_server.c: its existing index spawn now passes --index-worker so it does not re-supervise (one isolation layer, no redundant process nesting). - src/foundation/subprocess.c: the forked child redirects output with open()+dup2() instead of freopen() — async-signal-safe, no malloc between fork and exec, since the parent may be multithreaded (server threads + mimalloc/sqlite/libgit2 state). - internal/cbm/cbm.c: CBM_TEST_CRASH_ON / CBM_TEST_HANG_ON deterministic fault injectors (test-only, env-gated) so the guard is honest — green iff a real fault is genuinely contained, not a fixture that may stop faulting. - scripts/smoke-invariants.sh: inv_crasher_contained_cli asserts the crash escapes as a signal without the supervisor (rc>=128) and is contained + reported with it (rc<128, outcome=crash). Precise skip-and-continue (quarantine the culprit file, index the rest) is layered on in a follow-up; this change contains the crash and keeps the process alive. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…covery)
Stage 3b contained a crash to the worker but returned a failure response. Now the
supervisor pins the exact crashing file, quarantines it, and re-indexes the
healthy files: index_repository returns status="indexed" with the crasher listed
in skipped[] (phase="crash") and every other file indexed normally.
On a crash/hang the supervisor re-runs the worker single-threaded with a per-file
marker (CBM_INDEX_MARKER_FILE) so the marker names the EXACT crasher; that file is
appended to a quarantine list (CBM_INDEX_QUARANTINE_FILE) and the worker is
re-spawned until a clean run indexes the good files. Bounded by
CBM_INDEX_MAX_RESTARTS (default 100); each crash quarantines one new file, so the
loop makes monotonic progress.
- internal/cbm/cbm.c/.h: per-file marker + a lazily-loaded crash-quarantine set,
with a HARD guard at the top of cbm_extract_file (the one choke point every
extract pass funnels through) that short-circuits a quarantined file to an empty
result BEFORE the marker/parser — so no re-extracting pass (sequential
pass_calls/usages/semantic on a cache miss) can crash on it again.
- src/pipeline/pipeline.c: effective_worker_count() honours CBM_INDEX_SINGLE_THREAD
(=1 worker, sequential path) for exact single-threaded attribution.
- src/pipeline/pass_parallel.c, pass_definitions.c: report a quarantined file as a
phase="crash" skip via the Stage-2 skip plumbing so it surfaces in skipped[].
- src/mcp/index_supervisor.{h,c}: cbm_index_spawn_worker gains single_thread /
marker / quarantine knobs, passed to the child as inherited env around the
sequential spawn.
- src/mcp/mcp.c: index_run_supervised recovery loop (attribute -> quarantine ->
re-run until clean; else a contained-failure response).
- scripts/smoke-invariants.sh: inv_crasher_skipped_cli upgrades the 3b guard to the
skip-continue contract (baseline crash escapes; supervised -> status indexed +
crasher phase=crash + good file indexed + good file NOT skipped).
In-run quarantine only (not persisted across separate index runs); content-hash
persistence + re-admit is a follow-up.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… fallback
Completes the crash/hang supervisor. A file that makes the indexer make NO
progress (an external-scanner infinite loop, not a crash) is now contained the
same way a crash is: the supervisor's no-progress quiet-timeout kills the worker,
classifies it as a hang, and the recovery loop quarantines the marker-pinned file
and re-runs until a clean run indexes the good files.
- src/mcp/index_supervisor.c: worker_quiet_timeout_ms() defaults to a generous
15 min NO-PROGRESS timeout (reset by the worker's periodic log heartbeat), with
the CBM_INDEX_WORKER_TIMEOUT_S override. (--progress deliberately NOT added: it
installs a replace-mode sink + carriage-return updates that would remove the
newline heartbeat the quiet-timeout relies on.)
- Quarantine entries now carry the phase ("rel\tphase"): a fault signal -> "crash",
a no-progress kill -> "hang". internal/cbm/cbm.c parses it (bare path tolerated,
defaults crash) and exposes cbm_index_quarantine_phase(); the extract loops
(pass_parallel.c, pass_definitions.c) report the real phase + reason in skipped[].
- src/mcp/mcp.c: recovery loop logs the outcome, and a best-effort-PARTIAL terminal
spawn — when the loop cannot converge but files were quarantined, one final
quarantine-only run yields a partial index (good files indexed, known-bad ones
skipped) + a high-severity index.supervisor.partial log, instead of a hard error.
- scripts/smoke-invariants.sh: inv_hanger_skipped_cli (hang twin of the crasher
guard — baseline rc=124 vacuity check; supervised -> status indexed + hanger
phase=hang + good file indexed). Hardened run_bounded to force-kill (SIGKILL) a
SIGTERM-catching busy-spin and to not block on the fallback fifo open — the hang
guard is the first invariant to actually hang and exposed this latent harness bug.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…variableScope) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- format the include trailing comments and wrap the over-long reap log line (clang-format-20 violations flagged by lint / lint) - allowlist subprocess.c's fork with justification (security-static): the supervisor primitive forks and execs immediately; no code runs in the forked image - move bin into the fork-child block alongside argv (cppcheck variableScope consistency) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This was referenced Jul 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes indexing resilient to the residual failure class that no in-process guard can catch (external-scanner memory corruption, aborts, OOM-kill, scanner infinite loops): index_repository now runs in a supervised worker subprocess. A file that hard-crashes or hangs the native indexer is pinned, quarantined, and skipped — the run completes with status="indexed", the culprit reported in skipped[] (phase="crash"/"hang"), and every healthy file indexed. Previously one such file killed the whole MCP server or CLI (the #668 class).
Four commits:
<self> cli --index-worker index_repository … --response-out <tmp>(argv flag, not env — no ambient misfire; no re-supervision; http_server's existing spawn unified onto the same layer). fork+exec only (no pthread_atfork in the tree). Clean exit passes the worker response through; spawn failure degrades in-process; CBM_INDEX_SUPERVISOR=0 kill switch. Fork-safe child redirection via open()+dup2() (no malloc between fork and exec).Verified: baseline crash escapes (rc=134) → supervised rc=0 with the crasher skipped and good files indexed (nodes present); hang contained the same way; subprocess suite 13/13 under ASan/UBSan; smoke battery green with the new guards.
Closes #668.