test(e2e): pin regression coverage for verified-but-unpinned CLI behaviours (EAI-8072) - #293
Conversation
…rs (EAI-8072) Add GREEN mock-lane cucumber scenarios protecting CLI behaviours that were manually verified correct during the walkthrough but had no scenario guarding them. All run on the no-GPU/no-network mock lane every PR and assert host-invariant contracts (exit code + stable output) verified against the real Linux binary. - config mutations: default engine/runtime set+clear, telemetry, permissions, set-engine target requirement, provider enable/disable, local-provider guard, and provider-key save failing without leaking the key - logs: search match-count reporting, absent-term, service+search conflict - automations: enable/disable watcher mode confirmation, unknown-watcher refusal (background daemon spawn suppressed via a planted live-pid runtime state) - runtime lifecycle: activate/rollback/uninstall/import over planted read-only tarball runtimes (no SDK download or GPU needed) - update: report distinguishes configured from not-configured feeds (run with no runtimes so it stays offline) - networking: public-bind refused pre-flight without --allow-public-bind - serve: --runtime-id/--env-id mutually-exclusive selector conflict Adds a run_rocm_with_stdin helper for stdin-driven commands. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…ertions - Remove private internal workflow identifiers from public comments and fixture tokens (AGENTS.md permits EAI ids only): use EAI-8072 / neutral e2e tokens. - Make the provider-key no-secure-storage premise deterministic: force the Linux Secret Service unreachable via a bogus DBUS_SESSION_BUS_ADDRESS and scope the scenario @requires-os:linux, so it no longer relies on the runner happening to lack a session bus (Windows/macOS credential stores cannot be disabled the same way). Still verifies the key is never echoed. - Bind each update-feed status to its own feed line so a status attributed to the wrong feed fails, instead of matching feed names and statuses as independent substrings anywhere in the output. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Build the planted log body with writeln! (std::fmt::Write) instead of push_str(&format!(...)), and hoist the import to the top to avoid items_after_statements. Matches CI's pedantic/nursery clippy config, which the per-package check did not surface. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
5953765 to
98ad79d
Compare
tomastola
left a comment
There was a problem hiding this comment.
The new runtime lifecycle fixtures are not portable to Windows.
plant_runtime writes registry entries with:
registry.join(format!(\"{key}.json\"))The keys contain : (for example, therock-release:gfx942). On Windows, that creates an alternate data stream rather than a normal registry file, so the CLI cannot discover the planted runtime. The Windows E2E lane consequently has three unexpected failures:
runtime-activate-records-previousruntime-rollback-returns-to-previousruntime-uninstall-keeps-external-folder
All fail with installed runtime not found: therock-release:gfx942.
Please create these fixtures through a platform-safe path—preferably rocm runtimes import, which already passes on Windows and preserves the suite's black-box design—or use the same platform-safe filename encoding as production. These scenarios should not simply be skipped on Windows because the covered runtime operations are intended to be portable.
The runtime registry fixtures wrote entries at `<runtime_key>.json` using a `therock-release:gfx942`-style key. The `:` names an NTFS alternate data stream on Windows, so the planted file is not a discoverable registry entry and the activate/rollback/uninstall scenarios failed on the Windows E2E lane with "installed runtime not found". Use production-shaped keys: production derives `runtime_key` by slugifying, so a real key is filename-safe on every OS. Switch the fixture keys to the slugified form (`release-tarball-gfx942`) and keep the `therock-release:<family>` identifier only in `runtime_id`, which is a manifest field value and never a filename. Add an assertion in the planting helper so a future key that isn't filename-safe fails loudly rather than silently breaking one platform. Keeps the black-box hand-planting design; scenarios stay green on Linux and now create ordinary, discoverable registry files on Windows. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
|
Thanks @tomastola — good catch, fixed in c05b931. Root cause was exactly as you described: the fixtures wrote registry entries at The fix uses production-shaped keys. Production derives Verified the four runtime lifecycle scenarios still pass on Linux; the Windows E2E lane on this push should now find the planted runtimes. |
tomastola
left a comment
There was a problem hiding this comment.
Fix confirmed — keys now match production's slugify("{channel}-{format}-{family}") shape rather than a hand-picked safe-looking string, and the guard in plant_runtime keeps it that way. Windows lane green. Thanks.
Summary
Adds end-to-end regression scenarios that pin CLI behaviours which were manually verified as correct during the walkthrough but had no automated test guarding them. Everything here runs on the no-GPU/no-network mock lane (every PR) and asserts host-invariant contracts (exit code + stable output), so a regression surfaces in CI rather than in the field. Contracts were grounded by running the real Linux
rocmbinary, not read from source.Relates to EAI-8072 (linked from EAI-8024, EAI-7409, EAI-7404).
Changes
New feature files and step definitions:
set-enginerequiring a target; provider enable/disable; the local provider not being toggleable as a cloud provider; and saving a provider key failing without leaking the key when secure storage is unavailable.--searchreporting the matching-line count, an absent term reporting no matches, and the--service/--searchconflict being refused.--replace. Driven with planted read-only runtimes, so no SDK download or GPU is needed.--allow-public-bindis refused pre-flight (the EAI-7409 contract).--runtime-idand--env-idare rejected as a mutually-exclusive selector conflict.Also adds a small
run_rocm_with_stdintest helper (stdin + extra env) for commands that read from stdin.Test plan
engine-shell-marks-the-prompthost-environment failure, which also fails on unmodifiedmainin the same container and is green on CI's runner.cargo clippy --testsclean under-D warnings.Scope
This PR covers the mock-lane (no-GPU) regression set. The remaining GPU-only behaviours from the ticket (public-bind live auth/401, serve selector effects reflected in the serve plan,
--tool-call-parser,--no-smoke-test) require a live GPU serve to validate and are tracked separately.