ci(e2e): refresh the shared pre-warmed runtime when a newer one exists (EAI-8057)

[tomastola]

Summary

Nearly every GPU E2E scenario serves against one shared, pre-warmed managed
runtime, so a multi-GiB rocm install sdk happens once per runner instead of
once per scenario. That tree lives on the runner's persistent workspace and
survives git clean.

Cause. Every self-hosted lane guarded the install on directory existence
alone:

if [ ! -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then … fi

A persistent directory plus an existence-only guard means that branch runs
exactly once in the lifetime of a runner. After the first run ever, every
subsequent run served against whatever runtime happened to be installed that
day — 16 days old on both MI300X runners when measured. Two consequences: the
lanes validate an increasingly historical SDK, and drift between the shared tree
and what a fresh install sdk produces is never exercised at all, so a break in
the install path can only be discovered by a new runner.

Fix. Keep the cache, but invalidate it when the channel index has actually
published something newer. cargo xtask e2e-prewarm replaces the guard in all
eight self-hosted lanes and resolves to one of four outcomes:

Index vs. installed	Action
nothing installed for the channel	cold install sdk
index is ahead	install side-by-side, activate, prune
up_to_date / ahead_of_index	reuse
freshness unknown (index unreachable)	reuse, warn

Why it is shaped this way

Reuse the CLI's primitives instead of reimplementing version resolution.
rocm update already reports per-runtime freshness against the channel index,
rocm update --apply --runtime <key> --activate already installs a newer runtime
side-by-side, and rocm storage remove-old-installs --keep N already bounds the
resulting multi-version cache. Comparing versions in workflow shell would have
duplicated logic that exists — and is tested — in the product. It also means the
GPU lanes now exercise update and storage on real hardware, which nothing
did before.

Side-by-side, never move the tree. install sdk bakes absolute paths into
the runtime manifest, so a runtime has to be created in its final location. This
is why the update path uses --apply --activate rather than installing
elsewhere and swapping.

In xtask, not in the workflows. The pre-warm block is duplicated across
eight jobs in two shells (bash on the Linux lanes, PowerShell on Strix Windows).
Inline logic would exist eight times in two languages and be untestable; the
Strix Windows lanes lose their PowerShell copy rather than gaining a second
implementation to keep in sync. Same reasoning as the existing xtask e2e.

Anything unclear reuses. Only a report that positively reads as "a newer
version exists for our channel" triggers work. An unreachable package index
surfaces per-runtime as status=error, which reuses and warns — a briefly
offline index must not turn a GPU lane red, nor start a multi-GiB download on
every run. Install and update failures do propagate; a failed prune only warns.

A scenario pins the output shape — partially, and the gap is worth naming.
The decision is parsed out of rocm update text, so a fixture can drift from the
renderer it imitates. The new runtime-update-reports-freshness scenario runs the
real command on hardware and asserts the shape the parser depends on: a
runtime <key> … line carrying channel= and a recognised status=. That covers
structural drift — a renamed prefix or a dropped channel= filters every line out
and makes the lane reinstall on every run, which is loud; a renamed status=
field fails the scenario outright.

What it does not cover is a rename of the value update_available. A runner
that is currently up to date reports up_to_date, the assertion still passes, and
the decision quietly falls through to "reuse" — the exact defect this PR fixes.
Catching that reliably would need a runner that happens to be stale. The real
cause is that this is an unversioned text contract with more than one parser:
rocmd reads the same field by hand in
update_output_reports_update_available, also against hand-written fixtures, so
it carries the same exposure today. A machine-readable form of update, or a
shared constant for the status values, would close both. That is product-code
work and deliberately out of scope here.

Test plan

Verified locally:

• cargo test --workspace --all-targets — 2215 passed, 0 failed. Includes 12 new
  unit tests covering every freshness verdict and the degraded paths.
• cargo clippy --workspace --all-targets -- -D warnings and
  cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings clean;
  cargo fmt --all --check clean.
• The parser's fixtures are captured verbatim from the built binary, not
  hand-written. That caught a real near-collision: the update_surfaces summary
  line is one character away from parsing as a runtime entry, and there is now a
  test pinning that it does not.
• Full orchestration driven against a stub rocm on the PATH, so the sequencing
  is verified without a multi-GiB download: update → update --apply --runtime <key> --activate → storage remove-old-installs --keep 2 --yes, with
  ROCM_CLI_{CONFIG,DATA,CACHE}_DIR correctly scoped to the pre-warm root.

What the GPU lanes on this PR actually did

Every self-hosted lane reached the new pre-warm and reported the same verdict:

pre-warm: reusing the shared release runtime (runtime is up to date with the channel index)

Two things follow, one of which contradicts what I expected:

• The rocm update output contract holds on hardware. The new
  runtime-update-reports-freshness scenario passed all three steps on Strix
  Halo/Windows against a real index — the parser's assumption about the report
  shape is confirmed, not just imitated by fixtures.
• The update path did not fire, and that is the correct verdict. I predicted
  the 16-day-old MI300X trees would take the update path. They did not, because
  the Release channel currently resolves nothing newer than what those runners
  already hold: therock_index_urls() tries the classic per-family
  whl/{family} index first, which still succeeds but tops out at ROCm 7.13.0
  (rocm install sdk: release channel never resolves ROCm 7.14+ (stuck on 7.13.0) #271, fix open in fix(therock): resolve ROCm releases from the current multi-arch pip index #272). up_to_date is an honest answer to a capped index,
  so reuse is right — but it means the install, update, activate and prune
  paths are still unexercised on hardware, and remain covered only by unit
  tests and the stubbed-binary run above.

Worth being plain about the consequence: on the Release channel as it resolves
today, this change is a no-op every run. Its value is that freshness is now
consulted at all — the previous guard would have kept a tree frozen for the life
of a runner no matter what the index published, whereas this self-corrects on the
first run after the index moves. That is also why it has to land before the
nightly-channel follow-up: therock-nightly does move daily, and pointing a lane
at a different channel under an existence-only guard would have been a silent
no-op.

CI status

All required checks are green. Two self-hosted lanes are not, and neither is
caused by this change:

E2E tests (Strix Halo, Windows) — red, pre-existing. 8 of 41 scenarios
fail here against 8 of 40 on main, and the two failure sets are identical; the
+1 is the new scenario, which passes. That lane has failed on every main run
since 2026-08-12 (the visible cause is a llama-server.exe download failing
after 6 attempts); tracked in #260 and #247.

E2E tests (Strix Halo, Ubuntu) — cancelled at its 35-minute cap. The two
runners behind that label are not equivalent: strix-halo-ubuntu finishes in
13.6–15.9 min, while strix-halo-ubuntu-2 takes 31.0–31.5 min when it passes and
has hit the cap three times today — the other two on unrelated branches,
including main's own merge queue. This run drew runner-2. The pre-warm was not
the cost: it reused, downloaded nothing, and every serve scenario on that box ran
1.5–20× slower than on runner-1, including a refusal-path scenario that never
serves at all (371s vs under 20s).

One honest caveat on that second point: this PR does add a scenario to a lane
that, on runner-2, already sits within ~3.5 min of its cap, and the new
scenario's shared a managed runtime is active precondition is expensive on
that box specifically (0.6s on MI300X; still running at 140s on runner-2 when
the job was killed). So it is a small marginal push on an already-overcommitted
budget. The runner-2 disparity looks worth its own issue rather than a timeout
bump smuggled into this PR — happy to raise one, or to widen the cap here if a
maintainer would rather see that.

Risk

Low-to-moderate, and concentrated in one place: the lanes can now download.
Previously the pre-warm was a no-op after a runner's first run; now a genuinely
newer index costs one install before the suite. That is the intended behaviour,
but it does mean a lane can be slower than it was, and the conservative bias
exists so this happens only when the index has actually moved. In practice, on
the Release channel today, it has not — see above.

The failure modes are bounded the other way. Every ambiguous outcome reuses, so
the common bad day (index unreachable) behaves exactly like today. The lanes are
continue-on-error: true, so even a hard failure in the pre-warm cannot gate a
merge. No product code changes — this is CI orchestration, one new xtask
subcommand, and one new scenario.

Follow-up, tracked separately: with the tree no longer frozen, a scheduled lane
can track the therock-nightly channel, which is currently never installed by
CI (EAI-8056). That needs this first, for the reason given above.

• Searched tests/e2e-cucumber/expectations.toml for EAI-8057; no xfail
  rows reference it. The defect was in CI orchestration, not in any scenario's
  expected outcome, so there is nothing to narrow.

[r0x0r]

Reviewed — well-engineered and thorough. The existence-only guard genuinely froze the shared runtime after the first install ever, and moving the decision into xtask e2e-prewarm (once, cross-platform) instead of eight shell copies in two languages is the right call. Things I verified:

• keep defaults to 2 (default_value_t = 2), so the eight lanes calling e2e-prewarm --channel release --prewarm-dir … without --keep prune to 2, not 0 — no risk of pruning the active runtime.
• RuntimeLine::parse correctly rejects the update_surfaces runtimes: status=none_configured line (strip_prefix("runtime ") needs the trailing space), so an empty tree still resolves to Install rather than a spurious Reuse — and there's a dedicated test pinning exactly that near-collision.
• decide ordering is right: update_available wins over up_to_date/ahead_of_index, per-channel filtering keeps a release runtime from satisfying a nightly lane, and every degraded/unknown path is conservative (Reuse, never a red lane or a multi-GiB reinstall on a network blip). The probe-failed fallback in run preserves the old install-only-if-registry-absent floor.
• Env scoping (ROCM_CLI_{CONFIG,DATA,CACHE}_DIR → prewarm_dir) matches what the lanes exported by hand, and resolve_rocm_binary absolutizes a relative ROCM_CLI_BINARY like e2e.rs.
• 12 unit tests over verbatim-captured fixtures plus the on-hardware runtime-update-reports-freshness scenario.

One substantive thread on the parsing contract below — the gap you already named in the description; filing it as a resolvable thread for tracking rather than as a change request.

[r0x0r]

This routes on the literal string value update_available. As you call out in the PR body, the scenario pins the structure (a runtime <key> … channel=… status=… line) but not this value — if the renderer renamed update_available, the assertion still passes, this find misses, and the pre-warm silently falls through to Reuse, i.e. the exact staleness defect the PR fixes. And rocmd's update_output_reports_update_available hand-reads the same field against its own fixtures, so there are now (at least) two independent parsers of an unversioned text contract. Since both parsers already exist, would a single shared const for the status values — referenced by the renderer, this function, and rocmd — be cheap enough to land now and close both exposures at once? Understood if you'd rather keep it out of scope; capturing it here so it isn't lost.