Skip to content

test: stop process-global test state from failing unrelated tests - #208

Open
jmagar wants to merge 1 commit into
mainfrom
fix/test-path-override-isolation
Open

test: stop process-global test state from failing unrelated tests#208
jmagar wants to merge 1 commit into
mainfrom
fix/test-path-override-isolation

Conversation

@jmagar

@jmagar jmagar commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

cargo test intermittently failed 7–9 tests in inventory, setup::doctor, and agent_observatory — modules unrelated to whatever was being changed — with the failing set varying run to run. CI never saw it, because CI runs cargo nextest, which gives every test its own process. Three separate causes, all process-global state.

Closes syslog-mcp-g4frk.

1. A PATH override that leaked

container_probe_reports_unreachable_when_docker_ps_fails (added in #205) replaced PATH with a bare tempdir and never restored it. Every later bare-name spawn in the binary then resolved against a since-deleted directory and failed ENOENT.

This reproduces at --test-threads=1, so it is a leak, not a race#[serial] alone would not have fixed it, since #[serial] only excludes other #[serial] tests and the override outlives the test either way.

Now prepends to PATH instead of replacing it, restores through the file's own EnvGuard, and is #[serial] so it cannot collide with the three docker-stubbing tests in setup_tests.rs.

2. PATH replacement had no scoped alternative

collect_warns_when_optional_device_commands_are_missing legitimately wiped PATH to assert ip/ss/df are not installed — and in doing so stopped every concurrently running test from finding sh, git, and everything else.

This is why "fall back to the real PATH when the override misses" is not a viable fix: it breaks that test's intent. Instead this adds env::mask_test_programs, which makes named programs resolve to a path that cannot exist, so they spawn with the same NotFound an uninstalled binary produces. The test now states what it means, its blast radius is four program names instead of every program, and no test in the tree replaces PATH any more.

3. Projector tests asserted per-cycle values

oversized_first_rows=1 and "attempts":2 are both overwritten by the projector's next cycle, so each was true for roughly one 10 ms window and false forever after. No timeout could fix that — and notify_projection_work broadcasts on a process-global channel that every insert_logs_batch in the binary rings, so any test can drive any other test's projector through another cycle at any moment.

They now assert monotone facts: attempts >= 2, the durable cursors, and that health reports the counter rather than what it currently reads. The forward-progress property those tests exist for is still covered by the cursor assertion — with projector_page_bytes = 1 and a 16-byte row, a stalling page guard never advances the cursor.

Deadlines also moved to named constants documenting why they are generous: every test pool's init_pool runs the full migration set — VACUUM, CREATE INDEX, ANALYZE — while holding the one process-wide SQLite write lock.

Verification

baseline 79adf1f4 this branch
cargo test --no-fail-fast 8 failures 2471 passed, 0 failed
cargo nextest run green 3061 passed, 0 failed

Same machine, back to back. cargo clippy --all-targets and cargo fmt --check clean; lefthook pre-commit gates pass.

No production behaviour changes — every edit is test code or #[cfg(any(test, feature = "test-support"))].

`cargo test` intermittently failed 7-9 tests in inventory, setup::doctor, and
agent_observatory that had nothing to do with whatever was being changed. The
failing set varied run to run, and CI never saw any of it, so the failures read
as "your branch broke something" to whoever hit them. Three separate causes, all
process-global state that `cargo nextest` hides by giving each test its own
process.

PATH override leaked. `container_probe_reports_unreachable_when_docker_ps_fails`
(#205) replaced PATH with a bare tempdir and never restored it, so every later
bare-name spawn in the binary resolved against a deleted directory and failed
ENOENT. This reproduces at --test-threads=1, so it is a leak, not a race, and
`#[serial]` alone would not have fixed it. Now prepends and restores via the
file's own EnvGuard, and is `#[serial]` so it cannot collide with the three
docker-stubbing tests in setup_tests.rs.

PATH replacement had no scoped alternative. `collect_warns_when_optional_device_
commands_are_missing` legitimately wiped PATH to assert `ip`/`ss`/`df` are not
installed — which took every concurrent test's spawns down with it. Adding
`env::mask_test_programs`, which makes named programs resolve to an absent path
so they spawn with the same NotFound an uninstalled binary gives. The test now
says what it means, and no test in the tree replaces PATH any more.

Projector tests asserted per-cycle values. `oversized_first_rows=1` and
`"attempts":2` are overwritten by the projector's next cycle, so each was true
for roughly one 10ms window and false forever after — no timeout could fix
that, and `notify_projection_work` broadcasts on a process-global channel that
any test's `insert_logs_batch` rings, forcing extra cycles. They now assert
monotone facts: attempts >= 2, durable cursors, and that health reports the
counter rather than what it currently reads. Deadlines also moved to named
constants documenting the shared write lock, since every test pool's
`init_pool` migrates while holding it.

Verified on this tree: `cargo test --no-fail-fast` 2471/2471 (baseline
79adf1f: 8 failures) and `cargo nextest run` 3061/3061.

Refs: syslog-mcp-g4frk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant