test(e2e): pin the driver-as-root and comfyui-runtime contracts (EAI-8071) - #291
Open
fredespi wants to merge 8 commits into
Open
test(e2e): pin the driver-as-root and comfyui-runtime contracts (EAI-8071)#291fredespi wants to merge 8 commits into
fredespi wants to merge 8 commits into
Conversation
…EAI-8071) Two test-only scenarios pinning walkthrough defects, each registered as an xfail in expectations.toml citing its ticket so CI stays green and the row goes stale the day the bug is fixed. EAI-8053: `install driver --dkms` prefixes every plan command with a literal `sudo`, so as root on a host without sudo the first command dies with `sudo: not found`. New driver.feature runs the install with a PATH that omits sudo and asserts the run does not fail merely because sudo is absent. Adds a `@requires-root` tag (skips off root, where the prefix is correct) so the row stays honest on non-root lanes. EAI-8051: `comfyui install` installs ComfyUI's deps into the managed ROCm runtime with no index scoping, so a transitive dep can pull CUDA nvidia-* wheels in and displace the runtime's ROCm torch. New comfyui.feature installs an isolated runtime, installs ComfyUI, and asserts the runtime's torch is still a ROCm build with no nvidia-* distributions. GPU/lifecycle/nightly and isolated, since it may corrupt the runtime it runs against. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-eai-8051-and-eai-8053 Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/expectations.toml
Address pre-PR review of the ComfyUI runtime-preservation scenario: - Drop @lifecycle from the scenario. No lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so @requires-gpu @lifecycle @nightly together made the scenario unreachable on every lane. @requires-gpu @nightly matches runtime-install-sdk-active, the other scenario that does a real install sdk; this scenario mutates only its own isolated runtime prefix, not the OS. - Enumerate installed distributions via importlib.metadata instead of `python -m pip list`. uv-created managed runtimes have no pip module, so the pip probe exited non-zero with empty stdout, which the old code read as "no nvidia packages" and passed the check on a runtime it never inspected. The probe now fails loudly on any error rather than treating it as a clean result. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
CI's clippy lane checks the e2e harness test target with -D warnings, which surfaced two lints the mock-lane gate did not: - comfyui_steps.rs: map(str::trim).unwrap_or_else(panic) -> map_or_else. - e2e.rs: split run_rocm_with_only_tools' overlong first doc paragraph. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The helper and its which_on_path were #[cfg(unix)], but driver_steps.rs calls them unconditionally — cucumber step functions are registered on every platform, so the Windows build failed with E0425 (cannot find function). The driver scenario is @requires-os:linux and never runs on Windows, but the step code must still compile there. Drop the cfg gates and gate only the symlink line (Unix symlink; a plain copy on other platforms), so the call resolves everywhere. Verified: Linux clippy --all-targets -D warnings and a Windows cross-check (x86_64-pc-windows-gnu) both pass. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The EAI-8051 scenario located the managed runtime's venv by parsing `Folder:` from `rocm examine`, but examine only prints that line for the ACTIVE runtime and takes a different branch otherwise. On the MI300X GPU lane the baseline step panicked with "no active-runtime Folder: line" — a miswired step, reported as an unexpected failure rather than the intended xfail. The isolated `install sdk` itself succeeded. Read `install_root:` from `rocm runtimes list`, which is printed unconditionally for every installed runtime, so discovery no longer depends on which runtime examine considers active. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The EAI-8051 baseline step imported torch in the runtime venv to read torch.version.hip, but importing torch loads the ROCm shared libraries, which need the runtime's LD_LIBRARY_PATH/ROCM_PATH — the product sets that env when it probes torch, our bare interpreter does not. On the MI300X lane the import failed and a runtime with a perfectly good ROCm torch read as "not a ROCm build". Judge the build from the torch distribution's version label via importlib.metadata (`2.x+rocm...` vs `2.x+cu...`) instead — no import, no native load, so a bare interpreter suffices. The baseline and post-install assertions now also surface the actual version string for self-diagnosis on failure. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…nner The GPU lane showed TheRock's managed torch labels its local version with a git hash (2.11.0+gitd0c8b1f), not +rocm, so requiring "rocm" in the label wrongly failed the baseline on a healthy ROCm runtime. A CUDA wheel, by contrast, always carries +cu. The EAI-8051 corruption replaces the ROCm/TheRock torch with a CUDA one, so judge "still ROCm" as "torch present and NOT a +cu build" — which is exactly the flip the defect causes and needs no positive ROCm label. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two more walkthrough defects get pinning coverage, completing the set alongside EAI-8024 (#241) and EAI-8050. Test-only — no product behaviour changes. Each scenario asserts the correct behaviour and is registered as an expected failure in
expectations.tomlciting its ticket, so CI stays green and the row becomes a stale XPASS the day the bug is fixed.Pinned as expected failures
install driver --dkmsas root on a host withoutsudofails withsudo: not found, because the plan prefixes every command withsudoeven when already rootcomfyui installcan pull CUDAnvidia-*wheels into the managed ROCm runtime, displacing its ROCm torch — installing an optional app breaks the base runtimeEAI-8053 — driver install as root
New
driver.featurerunsinstall driver --dkms --yeswith a PATH that exposesshbut notsudo, and asserts the run does not fail merely becausesudois absent (not that the install succeeds — it can't in a container). A new@requires-roottag skips the scenario off root, where prefixingsudois correct, so the row stays honest on non-root lanes. Harness additions are confined to the test crate: arun_rocm_with_only_toolshelper (a PATH exposing only named tools) and the@requires-rootresolver support (uid read from/proc/self/status, nounsafe).EAI-8051 — comfyui must not destroy the ROCm runtime
New
comfyui.feature(the suite's first ComfyUI coverage) installs a managed runtime, installs ComfyUI, and asserts the runtime's torch is still a ROCm/HIP build with nonvidia-*distributions.@requires-gpu @nightly, run against the scenario's own isolated runtime prefix — never a shared one, since it may corrupt the runtime it checks. It does not assert the install exits 0 (the real install exits non-zero and still leaves the damage). Package enumeration usesimportlib.metadata(uv runtimes have nopip) and fails loudly rather than read a broken probe as clean.Test plan
sudo: not found); EAI-8051 skips as@nightly @requires-gpu.cargo test -p e2e-cucumber --libandcargo clippy -p e2e-cucumber --lib --tests -- -D warningspass in a Linux container.