Skip to content

test(e2e): pin regression coverage for verified-but-unpinned CLI behaviours (EAI-8072) - #293

Merged
fredespi merged 4 commits into
mainfrom
e2e-regression-coverage-for-walkthrough
Aug 25, 2026
Merged

test(e2e): pin regression coverage for verified-but-unpinned CLI behaviours (EAI-8072)#293
fredespi merged 4 commits into
mainfrom
e2e-regression-coverage-for-walkthrough

Conversation

@fredespi

Copy link
Copy Markdown
Collaborator

Summary

Adds end-to-end regression scenarios that pin CLI behaviours which were manually verified as correct during the walkthrough but had no automated test guarding them. Everything here runs on the no-GPU/no-network mock lane (every PR) and asserts host-invariant contracts (exit code + stable output), so a regression surfaces in CI rather than in the field. Contracts were grounded by running the real Linux rocm binary, not read from source.

Relates to EAI-8072 (linked from EAI-8024, EAI-7409, EAI-7404).

Changes

New feature files and step definitions:

  • config — set/clear default engine and runtime; telemetry and permissions modes; set-engine requiring a target; provider enable/disable; the local provider not being toggleable as a cloud provider; and saving a provider key failing without leaking the key when secure storage is unavailable.
  • logs--search reporting the matching-line count, an absent term reporting no matches, and the --service/--search conflict being refused.
  • automations — enabling/disabling a watcher confirms its mode; enabling an unknown watcher is refused. (The background daemon spawn is suppressed in-test so nothing leaks.)
  • runtime lifecycle — activate records the previous runtime, rollback returns to it, uninstall of an externally-sourced runtime keeps its folder, and import rejects a duplicate unless --replace. Driven with planted read-only runtimes, so no SDK download or GPU is needed.
  • update — the report distinguishes configured from not-configured update feeds (run with no managed runtimes so it stays offline).
  • networking — binding to a public interface without --allow-public-bind is refused pre-flight (the EAI-7409 contract).
  • serve--runtime-id and --env-id are rejected as a mutually-exclusive selector conflict.

Also adds a small run_rocm_with_stdin test helper (stdin + extra env) for commands that read from stdin.

Test plan

  • Full e2e cucumber suite green on a native-Linux container (mock lane): all new scenarios pass; reconciliation shows only the pre-existing engine-shell-marks-the-prompt host-environment failure, which also fails on unmodified main in the same container and is green on CI's runner.
  • cargo clippy --tests clean under -D warnings.
  • CI mock lane green on this PR.

Scope

This PR covers the mock-lane (no-GPU) regression set. The remaining GPU-only behaviours from the ticket (public-bind live auth/401, serve selector effects reflected in the serve plan, --tool-call-parser, --no-smoke-test) require a live GPU serve to validate and are tracked separately.

@fredespi
fredespi requested a review from a team as a code owner August 20, 2026 12:57
@fredespi
fredespi requested a review from tomastola August 20, 2026 12:57
…rs (EAI-8072)

Add GREEN mock-lane cucumber scenarios protecting CLI behaviours that were
manually verified correct during the walkthrough but had no scenario guarding
them. All run on the no-GPU/no-network mock lane every PR and assert
host-invariant contracts (exit code + stable output) verified against the real
Linux binary.

- config mutations: default engine/runtime set+clear, telemetry, permissions,
  set-engine target requirement, provider enable/disable, local-provider guard,
  and provider-key save failing without leaking the key
- logs: search match-count reporting, absent-term, service+search conflict
- automations: enable/disable watcher mode confirmation, unknown-watcher refusal
  (background daemon spawn suppressed via a planted live-pid runtime state)
- runtime lifecycle: activate/rollback/uninstall/import over planted read-only
  tarball runtimes (no SDK download or GPU needed)
- update: report distinguishes configured from not-configured feeds (run with no
  runtimes so it stays offline)
- networking: public-bind refused pre-flight without --allow-public-bind
- serve: --runtime-id/--env-id mutually-exclusive selector conflict

Adds a run_rocm_with_stdin helper for stdin-driven commands.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…ertions

- Remove private internal workflow identifiers from public comments and fixture
  tokens (AGENTS.md permits EAI ids only): use EAI-8072 / neutral e2e tokens.
- Make the provider-key no-secure-storage premise deterministic: force the Linux
  Secret Service unreachable via a bogus DBUS_SESSION_BUS_ADDRESS and scope the
  scenario @requires-os:linux, so it no longer relies on the runner happening to
  lack a session bus (Windows/macOS credential stores cannot be disabled the same
  way). Still verifies the key is never echoed.
- Bind each update-feed status to its own feed line so a status attributed to the
  wrong feed fails, instead of matching feed names and statuses as independent
  substrings anywhere in the output.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Build the planted log body with writeln! (std::fmt::Write) instead of
push_str(&format!(...)), and hoist the import to the top to avoid
items_after_statements. Matches CI's pedantic/nursery clippy config, which the
per-package check did not surface.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi force-pushed the e2e-regression-coverage-for-walkthrough branch from 5953765 to 98ad79d Compare August 20, 2026 13:31

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new runtime lifecycle fixtures are not portable to Windows.

plant_runtime writes registry entries with:

registry.join(format!(\"{key}.json\"))

The keys contain : (for example, therock-release:gfx942). On Windows, that creates an alternate data stream rather than a normal registry file, so the CLI cannot discover the planted runtime. The Windows E2E lane consequently has three unexpected failures:

  • runtime-activate-records-previous
  • runtime-rollback-returns-to-previous
  • runtime-uninstall-keeps-external-folder

All fail with installed runtime not found: therock-release:gfx942.

Please create these fixtures through a platform-safe path—preferably rocm runtimes import, which already passes on Windows and preserves the suite's black-box design—or use the same platform-safe filename encoding as production. These scenarios should not simply be skipped on Windows because the covered runtime operations are intended to be portable.

The runtime registry fixtures wrote entries at `<runtime_key>.json` using a
`therock-release:gfx942`-style key. The `:` names an NTFS alternate data stream
on Windows, so the planted file is not a discoverable registry entry and the
activate/rollback/uninstall scenarios failed on the Windows E2E lane with
"installed runtime not found".

Use production-shaped keys: production derives `runtime_key` by slugifying, so a
real key is filename-safe on every OS. Switch the fixture keys to the slugified
form (`release-tarball-gfx942`) and keep the `therock-release:<family>`
identifier only in `runtime_id`, which is a manifest field value and never a
filename. Add an assertion in the planting helper so a future key that isn't
filename-safe fails loudly rather than silently breaking one platform.

Keeps the black-box hand-planting design; scenarios stay green on Linux and now
create ordinary, discoverable registry files on Windows.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi

Copy link
Copy Markdown
Collaborator Author

Thanks @tomastola — good catch, fixed in c05b931.

Root cause was exactly as you described: the fixtures wrote registry entries at <runtime_key>.json using a therock-release:gfx942-style key, and the : names an NTFS alternate data stream on Windows, so the planted file was never a discoverable registry entry.

The fix uses production-shaped keys. Production derives runtime_key by slugifying (runtime_key() in therock.rs), so a real key is filename-safe on every OS. The fixtures now use the slugified form (release-tarball-gfx942) and keep the therock-release:<family> identifier only in runtime_id, which is a manifest field value and never a filename. This is the "same platform-safe filename encoding as production" path — it keeps the scenarios as GREEN black-box coverage of the portable runtime operations rather than skipping them on Windows. I also added an assertion in the planting helper so a future key that isn't filename-safe fails loudly instead of silently breaking one platform.

Verified the four runtime lifecycle scenarios still pass on Linux; the Windows E2E lane on this push should now find the planted runtimes.

@fredespi
fredespi requested a review from tomastola August 25, 2026 13:41

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix confirmed — keys now match production's slugify("{channel}-{format}-{family}") shape rather than a hand-picked safe-looking string, and the guard in plant_runtime keeps it that way. Windows lane green. Thanks.

@fredespi
fredespi added this pull request to the merge queue Aug 25, 2026
Merged via the queue into main with commit 16252c2 Aug 25, 2026
24 checks passed
@fredespi
fredespi deleted the e2e-regression-coverage-for-walkthrough branch August 25, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants