Skip to content

ci(e2e): refresh the shared pre-warmed runtime when a newer one exists (EAI-8057) #275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tomastola wants to merge 1 commit into
base: main
Choose a base branch
from
+676 −138
Open

ci(e2e): refresh the shared pre-warmed runtime when a newer one exists (EAI-8057) #275

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 35 additions & 77 deletions .github/workflows/e2e-selfhosted.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,8 +262,8 @@ jobs:
# unpacks an ~8.8 GiB devel tarball — is what blew the time cap). The
# "a managed runtime is active" precondition symlinks each scenario's
# data/runtimes here (see use_shared_runtimes); clean-slate scenarios
# stay isolated. Persisted across runs on RUNNER_WORKSPACE, so after the
# first run ever the pre-warm below is a no-op.
# stay isolated. Persisted across runs on RUNNER_WORKSPACE and refreshed
# by `xtask e2e-prewarm` when the channel index publishes a newer version.
#
# CRITICAL: the shared dir IS the pre-warm's own `data/runtimes`, and we
# NEVER move it. `install sdk` bakes ABSOLUTE paths (install_root,
Expand All @@ -284,27 +284,20 @@ jobs:

# Pre-warm the shared runtime ONCE, SERIALLY, before the suite — never
# lazily inside a concurrent scenario (two multi-GiB installs racing the
# same dir). Skipped once the tree is populated (it persists across runs).
# Uses the prebuilt binary (no cargo run --release) and the shared uv + HF
# caches. Installs directly into the persistent pre-warm data dir (no mv),
# so the manifest's absolute install_root stays valid for every scenario.
if [ ! -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "pre-warming shared runtime (first run on this runner)…"
mkdir -p "$prewarm"/{data,config,cache}
ROCM_CLI_CONFIG_DIR="$prewarm/config" \
ROCM_CLI_DATA_DIR="$prewarm/data" \
ROCM_CLI_CACHE_DIR="$prewarm/cache" \
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
"$ROCM_CLI_BINARY" install sdk
if [ -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "shared runtime pre-warmed at $E2E_SHARED_RUNTIMES_DIR"
else
echo "pre-warm did not produce a runtimes registry; scenarios will install their own" >&2
fi
else
echo "shared runtime already present at $E2E_SHARED_RUNTIMES_DIR — skipping pre-warm"
fi
# same dir). Installs directly into the persistent pre-warm data dir (no
# mv), so the manifest's absolute install_root stays valid for every
# scenario. Uses the prebuilt binary (no cargo run --release) and the
# shared uv + HF caches.
#
# This is a CACHE, not a one-shot: `xtask e2e-prewarm` installs when the
# tree is empty, installs the newer runtime side-by-side and activates it
# when the channel index has moved on, and otherwise reuses what is there.
# The old guard here tested directory existence only, so it never
# reinstalled and every lane froze on the first runtime it ever saw
# (16 days stale on both MI300X runners when measured — EAI-8057).
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

# Optional scenario-name filter for a scoped dispatch — lets a manual
# run select only the large-model scenario instead of the whole suite.
Expand Down Expand Up @@ -454,24 +447,12 @@ jobs:
export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

# Pre-warm once, serially, in place (no mv/symlink). Skipped once the tree
# is populated (persists across runs on RUNNER_WORKSPACE).
if [ ! -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "pre-warming shared runtime (first run on this runner)…"
mkdir -p "$prewarm"/{data,config,cache}
ROCM_CLI_CONFIG_DIR="$prewarm/config" \
ROCM_CLI_DATA_DIR="$prewarm/data" \
ROCM_CLI_CACHE_DIR="$prewarm/cache" \
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
"$ROCM_CLI_BINARY" install sdk
if [ -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "shared runtime pre-warmed at $E2E_SHARED_RUNTIMES_DIR"
else
echo "pre-warm did not produce a runtimes registry; scenarios will install their own" >&2
fi
else
echo "shared runtime already present at $E2E_SHARED_RUNTIMES_DIR — skipping pre-warm"
fi
# Pre-warm once, serially, in place (no mv/symlink), and refresh it when
# the channel index has moved on — the tree is a cache, not a one-shot.
# See the e2e-gpu lane and `xtask e2e-prewarm` for why the old
# existence-only guard froze this lane on its first runtime (EAI-8057).
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

# Optional scenario-name filter for a scoped manual dispatch.
NAME_FILTER="${{ github.event.inputs.name_filter }}"
Expand Down Expand Up @@ -615,24 +596,13 @@ jobs:
$env:ROCM_CLI_BINARY = "$targetDir\release\rocm.exe"
$env:ROCM_CLI_ROCMD_BINARY = "$targetDir\release\rocmd.exe"

# Pre-warm once, in place (no move/symlink). Skipped once the tree is
# populated (persists across runs on RUNNER_WORKSPACE).
if (-not (Test-Path "$env:E2E_SHARED_RUNTIMES_DIR\registry")) {
Write-Host "pre-warming shared runtime (first run on this runner)..."
New-Item -ItemType Directory -Force -Path "$prewarm\data","$prewarm\config","$prewarm\cache" | Out-Null
$env:ROCM_CLI_CONFIG_DIR = "$prewarm\config"
$env:ROCM_CLI_DATA_DIR = "$prewarm\data"
$env:ROCM_CLI_CACHE_DIR = "$prewarm\cache"
& $env:ROCM_CLI_BINARY install sdk
Remove-Item Env:\ROCM_CLI_CONFIG_DIR,Env:\ROCM_CLI_DATA_DIR,Env:\ROCM_CLI_CACHE_DIR -ErrorAction SilentlyContinue
if (Test-Path "$env:E2E_SHARED_RUNTIMES_DIR\registry") {
Write-Host "shared runtime pre-warmed at $env:E2E_SHARED_RUNTIMES_DIR"
} else {
Write-Host "pre-warm did not produce a runtimes registry; scenarios will install their own"
}
} else {
Write-Host "shared runtime already present at $env:E2E_SHARED_RUNTIMES_DIR - skipping pre-warm"
}
# Pre-warm once, in place (no move/symlink), and refresh it when the
# channel index has moved on — the tree is a cache, not a one-shot. The
# decision lives in `xtask e2e-prewarm` rather than being reimplemented
# here in PowerShell, so this lane and the Linux lanes cannot drift
# (EAI-8057; the old existence-only guard never reinstalled).
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }

# Optional scenario-name filter for a scoped manual dispatch.
$nameFilter = "${{ github.event.inputs.name_filter }}"
Expand Down Expand Up @@ -809,24 +779,12 @@ jobs:
export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

# Pre-warm once, serially, in place (no mv/symlink). Skipped once the
# tree is populated (persists across runs on RUNNER_WORKSPACE).
if [ ! -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "pre-warming shared runtime (first run on this runner)…"
mkdir -p "$prewarm"/{data,config,cache}
ROCM_CLI_CONFIG_DIR="$prewarm/config" \
ROCM_CLI_DATA_DIR="$prewarm/data" \
ROCM_CLI_CACHE_DIR="$prewarm/cache" \
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
"$ROCM_CLI_BINARY" install sdk
if [ -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "shared runtime pre-warmed at $E2E_SHARED_RUNTIMES_DIR"
else
echo "pre-warm did not produce a runtimes registry; scenarios will install their own" >&2
fi
else
echo "shared runtime already present at $E2E_SHARED_RUNTIMES_DIR — skipping pre-warm"
fi
# Pre-warm once, serially, in place (no mv/symlink), and refresh it when
# the channel index has moved on — the tree is a cache, not a one-shot.
# See the e2e-gpu lane and `xtask e2e-prewarm` for why the old
# existence-only guard froze this lane on its first runtime (EAI-8057).
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

# Optional scenario-name filter for a scoped manual dispatch.
NAME_FILTER="${{ github.event.inputs.name_filter }}"
Expand Down
81 changes: 20 additions & 61 deletions .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,19 +389,13 @@ jobs:
export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

# Pre-warm the shared runtime once/serially before the suite; skipped
# once populated (persists across runs). Install in place (no mv) so the
# manifest's absolute paths stay valid. See ci.yml e2e-gpu for rationale.
if [ ! -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "pre-warming shared runtime (first run on this runner)…"
mkdir -p "$prewarm"/{data,config,cache}
ROCM_CLI_CONFIG_DIR="$prewarm/config" \
ROCM_CLI_DATA_DIR="$prewarm/data" \
ROCM_CLI_CACHE_DIR="$prewarm/cache" \
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
"$ROCM_CLI_BINARY" install sdk
fi
# Pre-warm the shared runtime once/serially before the suite, refreshing
# it when the channel index has a newer version. Install in place (no mv)
# so the manifest's absolute paths stay valid. See the e2e-selfhosted.yml
# e2e-gpu lane and `xtask e2e-prewarm` for the full rationale (EAI-8057).
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

cargo xtask e2e

Expand Down Expand Up @@ -496,22 +490,10 @@ jobs:
export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

if [ ! -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "pre-warming shared runtime (first run on this runner)…"
mkdir -p "$prewarm"/{data,config,cache}
ROCM_CLI_CONFIG_DIR="$prewarm/config" \
ROCM_CLI_DATA_DIR="$prewarm/data" \
ROCM_CLI_CACHE_DIR="$prewarm/cache" \
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
"$ROCM_CLI_BINARY" install sdk
if [ -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "shared runtime pre-warmed at $E2E_SHARED_RUNTIMES_DIR"
else
echo "pre-warm did not produce a runtimes registry; scenarios will install their own" >&2
fi
else
echo "shared runtime already present at $E2E_SHARED_RUNTIMES_DIR — skipping pre-warm"
fi
# Cache, not a one-shot: refreshed when the channel index has a newer
# version. See `xtask e2e-prewarm` (EAI-8057).
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

cargo xtask e2e

Expand Down Expand Up @@ -604,22 +586,11 @@ jobs:
$env:ROCM_CLI_BINARY = "$targetDir\release\rocm.exe"
$env:ROCM_CLI_ROCMD_BINARY = "$targetDir\release\rocmd.exe"

if (-not (Test-Path "$env:E2E_SHARED_RUNTIMES_DIR\registry")) {
Write-Host "pre-warming shared runtime (first run on this runner)..."
New-Item -ItemType Directory -Force -Path "$prewarm\data","$prewarm\config","$prewarm\cache" | Out-Null
$env:ROCM_CLI_CONFIG_DIR = "$prewarm\config"
$env:ROCM_CLI_DATA_DIR = "$prewarm\data"
$env:ROCM_CLI_CACHE_DIR = "$prewarm\cache"
& $env:ROCM_CLI_BINARY install sdk
Remove-Item Env:\ROCM_CLI_CONFIG_DIR,Env:\ROCM_CLI_DATA_DIR,Env:\ROCM_CLI_CACHE_DIR -ErrorAction SilentlyContinue
if (Test-Path "$env:E2E_SHARED_RUNTIMES_DIR\registry") {
Write-Host "shared runtime pre-warmed at $env:E2E_SHARED_RUNTIMES_DIR"
} else {
Write-Host "pre-warm did not produce a runtimes registry; scenarios will install their own"
}
} else {
Write-Host "shared runtime already present at $env:E2E_SHARED_RUNTIMES_DIR - skipping pre-warm"
}
# Cache, not a one-shot: refreshed when the channel index has a newer
# version. The decision lives in `xtask e2e-prewarm` so this lane cannot
# drift from the Linux ones (EAI-8057).
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }

cargo xtask e2e

Expand Down Expand Up @@ -765,22 +736,10 @@ jobs:
export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

if [ ! -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "pre-warming shared runtime (first run on this runner)…"
mkdir -p "$prewarm"/{data,config,cache}
ROCM_CLI_CONFIG_DIR="$prewarm/config" \
ROCM_CLI_DATA_DIR="$prewarm/data" \
ROCM_CLI_CACHE_DIR="$prewarm/cache" \
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
"$ROCM_CLI_BINARY" install sdk
if [ -d "$E2E_SHARED_RUNTIMES_DIR/registry" ]; then
echo "shared runtime pre-warmed at $E2E_SHARED_RUNTIMES_DIR"
else
echo "pre-warm did not produce a runtimes registry; scenarios will install their own" >&2
fi
else
echo "shared runtime already present at $E2E_SHARED_RUNTIMES_DIR — skipping pre-warm"
fi
# Cache, not a one-shot: refreshed when the channel index has a newer
# version. See `xtask e2e-prewarm` (EAI-8057).
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

cargo xtask e2e

Expand Down
37 changes: 37 additions & 0 deletions docs/ci-hardware-testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,43 @@ Dispatch the GPU lanes with, e.g.:
gh workflow run e2e-selfhosted.yml --ref <ref> -f platform=app-dev-gpu
```

## The shared pre-warmed runtime

Nearly every GPU serve scenario points its `data/runtimes` at one shared,
pre-warmed managed runtime (`E2E_SHARED_RUNTIMES_DIR`), so a multi-GiB
`rocm install sdk` happens once per runner instead of once per scenario. The tree
lives on the runner's persistent workspace and survives `git clean`.

It is a **cache with invalidation**, not a one-shot install. Each self-hosted lane calls

```bash
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"
```

before the suite, which asks `rocm update` whether the channel index has published
a newer version and then:

- installs the SDK when nothing is present for that channel;
- installs the newer runtime **side-by-side** and activates it
(`rocm update --apply --runtime <key> --activate`) when the index is ahead;
- reuses the existing tree when it is `up_to_date`, when it is `ahead_of_index`
(a pinned build newer than the index must not be rolled back), or when freshness
cannot be established at all — an unreachable index reuses and warns rather than
re-downloading gigabytes or failing the lane;
- prunes with `rocm storage remove-old-installs` after any install or update, so
the multi-version cache stays bounded.

The runtime is always installed **in place**: `install sdk` bakes absolute paths
into the runtime manifest, so a tree that is moved after installation leaves every
serve pointing at a path that no longer exists.

This replaced an existence-only guard that never reinstalled, which had frozen both
MI300X runners on a 16-day-old runtime and left drift from a fresh install untested
(`EAI-8057`). The decision logic lives in `xtask` rather than the workflows because
the pre-warm block is duplicated across eight jobs in two shells; `xtask/src/e2e_prewarm.rs`
carries unit tests for each freshness verdict, and the `runtime-update-reports-freshness`
scenario pins the `rocm update` output shape those tests assume.

## Blocking vs. non-blocking

The four self-hosted jobs — `e2e-gpu`, `e2e-gpu-strix-ubuntu`,
Expand Down
15 changes: 15 additions & 0 deletions tests/e2e-cucumber/features/runtime_setup.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,21 @@ Feature: Runtime configuration
When the user inspects the system
Then the managed runtime folder path is not recursively nested

# The GPU E2E lanes no longer install the shared runtime once and keep it
# forever: `xtask e2e-prewarm` asks `rocm update` whether the channel index has
# published a newer version, and installs it side-by-side when it has (EAI-8057).
# That makes CI depend on the freshness line this scenario pins. A unit test on a
# hand-written fixture cannot catch the renderer drifting away from the parser —
# only running the real command can, which is why this is a scenario and not just
# an xtask test. Cheap enough for the per-PR lanes: one CLI call against the
# already-installed shared runtime. `status=error` is an ACCEPTED outcome, so an
# offline runner reports honestly instead of flaking.
@id:runtime-update-reports-freshness @requires-gpu
Scenario: 4 - The update check reports the active runtime's freshness
Given a managed runtime is active
When the user checks for runtime updates
Then the report states the runtime's freshness against the channel index

# Linux-only: the step adopts a standard `/opt/rocm` install with a Unix python
# path. On Windows those paths don't exist (the CLI resolves `/usr/bin/python3`
# to a bogus `C:/usr/bin/python3` and errors on the missing path before it can
Expand Down
Loading
Loading