First Version of E2E Validation Infrastructure

[Elliot (theelliotm)]

📖 Description

This PR introduces an E2E validation testing matrix that uses self-hosted runners to run tests across multiple OS versions. Currently includes support for:

x64

• Windows: Prerelease, 25H2, 24H2, 23H2
• MacOS: 26, 15
• Linux: Ubuntu 26.04, Ubuntu 24.04, Debian 13, RHEL 10

arm64
As of now, there are no Azure VMs that support nested virt on ARM CPUs.

Current Backend Support

• Process-T1: ✅ GOOD. Supported on prerelease Windows. Test failures are either a fault of the testing suite or legitimate issues with MXC.
• Process-T3: ⚠️✅ (MOSTLY) GOOD: Tests fail on versions that support T1 due to test suite limitations. Other test failures are either a fault of the testing suite or legitimate issues with MXC.
• Sandbox: ⛔ NOT IMPLEMENTED
• WSLC: ⚠️✅ (MOSTLY) GOOD. There is a chance of hitting a rate-limit when updating WSL and downloading WSLC images in the test script itself. Other test failures are either a fault of the testing suite or legitimate issues with MXC.
• Isolation Session: ⚠️ ISSUE: Feature_AgentSessionsBaseSupport is not enabled on the VM. Seeking internal feedback on how to enable this.
• Bubblewrap: ✅ GOOD.
• LXC: ⚠️✅ (MOSTLY) GOOD: Some networking tests fail on versions besides Ubuntu 24. I'm not sure if it's the VM's fault or an issue with MXC.
• MicroVM: ⛔ NOT WORKING: (Windows) Both cold and warm starts freeze and timeout. (Linux) No E2E test suite available.
• Hyperlight: ⛔ NOT IMPLEMENTED
• Seatbelt: ✅ GOOD. Currently no official E2E test suite, but a temporary one I had Copilot write didn't have any issues running or using Seatbelt APIs.

Schedule

The full test matrix runs nightly on main. Once we get bugs fixed and a clearer idea of cost, the schedule will be adjusted.

The schedule, testing matrix, and all self-hosted runners are fully configurable in /scripts/ci/validation-test-matrix.json.

🔗 References

🔍 Validation

No MXC code was modified. All newly created workflows can run and provide feedback on resolving issues.

✅ Checklist

• Signed the Contributor License Agreement
• Updated documentation (if applicable)
• Updated Copilot instructions (if build, architecture, or conventions changed)

📋 Issue Type

• Feature

Microsoft Reviewers: Open in CodeFlow

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Introduces artifact-driven E2E validation across Windows, Linux, and macOS runners.

Changes:

• Adds validation matrix resolution, scheduling, and backend dispatch.
• Adds host preparation, diagnostics, and temporary Seatbelt tests.
• Expands build artifacts and updates runner configurations.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/scripts/TEMP_run_seatbelt_all_tests.sh	Adds temporary Seatbelt E2E coverage.
tests/scripts/run_wslc_all_tests.ps1	Makes optional JSON reads StrictMode-safe.
tests/scripts/run_microvm_tests.ps1	Updates artifact checks and StrictMode handling.
tests/scripts/run_lxc_network_test.sh	Adds network diagnostics.
tests/scripts/run_ci_backend_tests.sh	Dispatches Unix backend tests.
tests/scripts/run_ci_backend_tests.ps1	Dispatches Windows backend tests.
tests/scripts/run_bwrap_denied_masking_test.sh	Corrects fixture ownership.
tests/scripts/README.md	Updates script scope documentation.
tests/configs/lxc_network_diagnostics.json	Defines LXC diagnostic execution.
scripts/ci/validation-test-matrix.json	Declares platforms, backends, and plans.
scripts/ci/resolve-validation-test-matrix.mjs	Validates and expands matrix plans.
scripts/ci/prepare-windows-host.ps1	Prepares Windows backend prerequisites.
scripts/ci/prepare-linux-host.sh	Prepares Linux backend prerequisites.
docs/pull-requests.md	Updates PR workflow documentation.
.github/workflows/Validation.Tests.Scheduled.yml	Schedules validation plans.
.github/workflows/Validation.Tests.Matrix.Job.yml	Runs resolved cross-platform tests.
.github/workflows/Validation.Infra.Testing.yml	Exercises enabled infrastructure entries.
.github/workflows/SDK.Integration.Test.Job.yml	Moves macOS integration tests to macOS 26.
.github/workflows/microvm-e2e.yml	Documents temporary workflow retention.
.github/workflows/Lint.Job.yml	Moves macOS linting to macOS 26.
.github/workflows/hyperlight-e2e.yml	Documents temporary workflow retention.
.github/workflows/Build.Windows.Job.yml	Expands and verifies Windows artifacts.
.github/workflows/Build.MacOS.Job.yml	Updates runner and verifies artifacts.
.github/workflows/Build.Linux.Job.yml	Adds features and artifact verification.
.github/workflows/Build.And.Validate.yml	Adds PR matrix orchestration.
.github/copilot-instructions.md	Documents validation infrastructure.
.github/actions/setup-cargo-feed/action.yml	Uses PowerShell for feed configuration.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

Suppressed comments (7)

tests/scripts/run_ci_backend_tests.ps1:119

• The IsolationSession suite exits 0 after printing SKIPPED when probes.isolationSessionAvailable is false, but this wrapper only checks $LASTEXITCODE. The scheduled matrix will therefore report a successful validation even when no IsolationSession test ran (which is the current pool state described by this PR). Treat the suite-level SKIPPED result as a failed/unavailable matrix job, or gate scheduling on a successful availability probe.

    'isolation-session' {
        Invoke-TestScript -Path (Join-Path $scriptRoot 'run_isolation_session_tests.ps1') -Arguments @{
            WxcExePath = $wxc
        }

.github/workflows/Build.Linux.Job.yml:96

• Commenting out this prerequisite silently removes Bubblewrap characterization coverage from normal PR builds: those tests explicitly skip when has_bwrap() is false, and the new pr validation plan is empty. Restore the install/setup step so cargo test -p wxc_e2e_tests continues exercising Bubblewrap before merge.

      # - name: Install Bubblewrap
      #   working-directory: ${{ github.workspace }}
      #   run: |
      #     sudo apt-get update
      #     sudo apt-get install -y bubblewrap
      #     # Ubuntu 24.04 runners restrict unprivileged user namespaces via
      #     # AppArmor, which blocks `bwrap --unshare-user`. Relax it so the
      #     # sandbox can start (no-op on kernels without this knob).
      #     sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true

.github/copilot-instructions.md:53

• All three referenced workflow paths are absent. The implemented entry point is Build.yml, scheduled jobs invoke the OS-specific build workflows directly, and the matrix workflow is Validation.Tests.Matrix.Job.yml. Correct these instructions so they describe the architecture actually added by this PR.

`.github/workflows/Build.And.Validate.yml` is the PR/CI entry point. It calls the
workflow-call-only `.github/workflows/Build.Artifacts.Job.yml`, which builds and
uploads the Windows, Linux, and macOS artifacts in parallel. Scheduled E2E and
validation-infrastructure workflows reuse the same artifact workflow before
calling `.github/workflows/Test.Matrix.Job.yml`; keep artifact production and
consumption in the same workflow run.

docs/pull-requests.md:8

• These workflow names do not exist in the repository. The PR entry point remains Build.yml, which directly invokes the OS-specific reusable build workflows; there is no Build.Artifacts.Job.yml. Documenting nonexistent entry points makes the contributor guidance unusable.

`.github/workflows/` (entry point: `Build.And.Validate.yml`). This is the primary
PR signal — it calls the reusable `Build.Artifacts.Job.yml` workflow, then
tests on native Windows x64/arm64, Linux x64/arm64, and macOS arm64 hosts.

.github/workflows/Build.Linux.Job.yml:81

• Test.Matrix.Job.yml does not exist in this repository; the workflow introduced by this PR is Validation.Tests.Matrix.Job.yml. Update the comment so future maintainers can locate the actual destination workflow.

This issue also appears on line 88 of the same file.

      # E2E Tests are being moved to the Test.Matrix.Job.yml workflow, which runs cross-platform validation

scripts/ci/resolve-validation-test-matrix.mjs:26

• The new catalog validator/matrix expander has no automated tests, despite exporting pure functions and implementing many failure and filtering branches. Add Node tests covering invalid catalogs, trigger expansion, empty plans, pool suppression, and deterministic sorting so matrix changes are validated without allocating self-hosted runners.

export function validateCatalog(catalog) {

.github/workflows/Validation.Infra.Testing.yml:6

• This infrastructure-validation workflow is permanently scoped to the PR author's branch. After merge, changes from any other branch will not exercise the enabled matrix before landing; only the post-merge schedule can discover breakage. Use a pull_request trigger with relevant path filters (and optionally workflow_dispatch), or remove this temporary workflow before merge.

  push:
    branches:
      - user/emichlin/validation-infra-lite

[Elliot (theelliotm)]

@microsoft-github-policy-service agree company="Microsoft"

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

Suppressed comments (13)

tests/scripts/run_ci_backend_tests.ps1:120

• This CI wrapper treats the suite's exit code 0 as a pass, but run_isolation_session_tests.ps1 exits 0 with SKIPPED when isolationSessionAvailable=false. Because the nightly pool is currently known to lack that feature, this job can report success without executing a test. Add a CI-required availability mode (or explicitly probe here and fail when unavailable) before counting this as validation.

    'isolation-session' {
        Invoke-TestScript -Path (Join-Path $scriptRoot 'run_isolation_session_tests.ps1') -Arguments @{
            WxcExePath = $wxc
        }
    }

.github/workflows/Validation.Tests.Matrix.Job.yml:43

• Pin this action to a full commit SHA, matching the established workflow convention (.github/workflows/Build.Windows.Job.yml:28). A mutable major-version tag can change code executed on the validation runner without a repository change.

      - uses: actions/checkout@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:101

• Pin this checkout action to the same full SHA used by the existing build workflows; @v4 is a mutable reference.

      - uses: actions/checkout@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:104

• Pin this download action to the full SHA already used elsewhere in the repository instead of the mutable @v4 tag.

        uses: actions/download-artifact@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:133

• Pin this upload action to the repository's existing full SHA instead of the mutable @v4 tag.

        uses: actions/upload-artifact@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:150

• Pin this checkout action to the full SHA used by the existing workflows; @v4 is mutable.

      - uses: actions/checkout@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:153

• Pin this download action to the repository's existing full SHA instead of the mutable @v4 tag.

        uses: actions/download-artifact@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:169

• Pin this upload action to the repository's existing full SHA instead of the mutable @v4 tag.

        uses: actions/upload-artifact@v4

tests/scripts/run_microvm_tests.ps1:98

• Do not require the WHP snapshots here. They are only a warm-start cache: the NanVix runner generates them when absent, and Initialize-MicroVmHost deliberately excludes them from the required artifact payload. Requiring them makes an otherwise valid downloaded artifact fail before the runtime fallback can run.

    "snapshots\kernel.vmem",
    "snapshots\kernel.whp.cbor"

scripts/ci/validation-test-matrix.json:350

• The PR says the full matrix covers macOS 15/26 nightly, but this plan has no macOS request and weekly is also empty, so the resolver never emits a Seatbelt job. Either wire and schedule Seatbelt coverage or narrow the stated scope; as written, the advertised macOS E2E coverage never runs.

    "nightly": [

.github/workflows/Validation.Tests.Matrix.Job.yml:27

• Pin this action to a full commit SHA. Repository workflows consistently pin actions/checkout (for example, .github/workflows/Build.Windows.Job.yml:28); the mutable @v4 tag unnecessarily exposes validation runners to upstream tag changes.

This issue also appears in the following locations of the same file:

• line 43
• line 101
• line 150

      - uses: actions/checkout@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:80

• Use the repository's SHA-pinned artifact uploader (.github/workflows/Build.Windows.Job.yml:104) rather than the mutable @v4 tag.

This issue also appears in the following locations of the same file:

• line 133
• line 169

        uses: actions/upload-artifact@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:46

• Use the repository's SHA-pinned artifact downloader (.github/workflows/SDK.Integration.Test.Job.yml:45) rather than the mutable @v4 tag.

This issue also appears in the following locations of the same file:

• line 104
• line 153

        uses: actions/download-artifact@v4

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.

Suppressed comments (12)

.github/workflows/Validation.Tests.Matrix.Job.yml:46

• Pin this artifact action to the full SHA used by the other workflows; the mutable tag can move to unreviewed code executed on the self-hosted runner.

        uses: actions/download-artifact@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:101

• Pin this checkout action to the repository's reviewed full SHA rather than a mutable tag, especially because this job runs on a self-hosted Linux runner.

      - uses: actions/checkout@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:104

• Pin this artifact action to the full SHA used by the other workflows; the mutable tag can move to unreviewed code executed on the self-hosted runner.

        uses: actions/download-artifact@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:133

• Pin this upload action to the repository's reviewed full SHA instead of the mutable v4 tag.

        uses: actions/upload-artifact@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:150

• Pin this checkout action to the repository's reviewed full SHA rather than a mutable tag, matching the rest of the workflow set.

      - uses: actions/checkout@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:153

• Pin this artifact action to the full SHA used by the other workflows instead of a mutable tag.

        uses: actions/download-artifact@v4

.github/workflows/Validation.Tests.Matrix.Job.yml:169

• Pin this upload action to the repository's reviewed full SHA instead of the mutable v4 tag.

        uses: actions/upload-artifact@v4

scripts/ci/validation-test-matrix.json:357

• This suite treats isolationSessionAvailable=false as SKIPPED and exits 0, while the PR description says this pool does not have Feature_AgentSessionsBaseSupport enabled. Consequently this nightly entry reports success without executing any IsolationSession test. Remove the entry until the pool supports the backend, or make the CI wrapper reject an availability skip.

          "isolation-session",

scripts/ci/validation-test-matrix.json:410

• The scheduled workflow invokes the weekly plan every Sunday and also offers it for manual dispatch, but this empty plan resolves to no backend jobs and succeeds after only the resolver runs. Populate the plan or remove the weekly schedule/option until it has coverage so the weekly validation signal is not a no-op.

    "weekly": [],

scripts/ci/validation-test-matrix.json:350

• The PR description says the full matrix, including macOS 26/15 Seatbelt coverage, runs nightly, but this plan contains no macOS trigger and the Seatbelt dispatcher always exits 2. Thus the scheduled workflow builds macOS artifacts but never runs a macOS E2E test. Wire and trigger Seatbelt coverage, or update the stated scope.

    "nightly": [

tests/scripts/run_microvm_tests.ps1:97

• These snapshots are an optional warm-start cache: the NanVix build/runtime explicitly supports them being absent, and prepare-windows-host.ps1 intentionally excludes them from required artifact files. Requiring them here makes a valid cold-start artifact fail before the runtime can regenerate the cache.

    "snapshots\kernel.vmem",
    "snapshots\kernel.whp.cbor"

docs/pull-requests.md:5

• This still says every PR is validated, but Build.yml now ignores docs/**, so a documentation-only PR does not trigger this workflow. Qualify the statement or remove the path exclusion.

`.github/workflows/` (entry point: `Build.yml`). This is the primary PR signal —

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

tests/scripts/run_ci_backend_tests.ps1:120

• An unavailable IsolationSession exits this script with code 0 at run_isolation_session_tests.ps1:200-203, and because it is invoked in-process, that exit terminates this wrapper successfully before any post-check. On the currently described VM lacking Feature_AgentSessionsBaseSupport, the nightly job is therefore green with zero tests; CI must treat an unavailable backend as a failed prerequisite (or explicitly inspect and reject the skip).

    'isolation-session' {
        Invoke-TestScript -Path (Join-Path $scriptRoot 'run_isolation_session_tests.ps1') -Arguments @{
            WxcExePath = $wxc
        }
    }

scripts/ci/validation-test-matrix.json:350

• The PR says the full matrix runs nightly and lists macOS 26/15 with Seatbelt as supported, but the nightly plan contains no macOS request, so has_macos is always false and both platforms are skipped. Moreover, the Seatbelt dispatcher currently exits 2. Either wire and schedule Seatbelt coverage or update the claimed scope; as written, this infrastructure provides no macOS E2E validation.

  "triggers": {
    "pr": [],
    "nightly": [

[Copilot]

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Suppressed comments (4)

scripts/ci/validation-test-matrix.json:356

• This nightly entry can silently pass without testing IsolationSession. The suite exits 0 with SKIPPED when isolationSessionAvailable=false (run_isolation_session_tests.ps1:200-203), while the new dispatcher checks only $LASTEXITCODE (run_ci_backend_tests.ps1:46-52). Because the pool currently lacks the required feature, remove this trigger until the host is enabled or make CI treat that skip as a failed prerequisite.

          "isolation-session",

scripts/ci/validation-test-matrix.json:350

• The PR says the full matrix runs nightly and lists macOS 15/26 with Seatbelt support, but nightly contains no macOS request. Consequently the scheduled workflow builds the macOS artifact but has_macos is false and no macOS validation job runs; moreover the Seatbelt dispatcher currently exits as unwired. Wire a Seatbelt suite and add both macOS platforms, or narrow the stated scope.

    "nightly": [

sdk/node/tests/integration/test-helpers.ts:52

• Adding these test executables to the package expectations makes them part of the published npm payload, not just the E2E artifact: Package.NpmSdk.Job.yml downloads the entire Windows artifact into sdk/node/bin, and package.json publishes all of bin/. Split/filter the validation-only binaries instead of requiring wxc-ui-probe.exe and wxc-test-driver.exe in the public SDK package.

  'wxc-ui-probe.exe',
  'wxc-test-driver.exe',

docs/pull-requests.md:10

• This now overstates coverage: Build.yml:9-12 ignores pull requests whose only changes are under docs/**, so those PRs do not run this entry point. Qualify the statement (or remove the path filter) so contributors are not told every PR receives these build and test signals.

`.github/workflows/` (entry point: `Build.yml`). This is the primary PR signal —
it fans out to the reusable `Build.Windows.Job.yml`, `Build.Linux.Job.yml`, and
`Build.MacOS.Job.yml` workflows, which build and test on native Windows
x64/arm64, Linux x64/arm64, and macOS arm64 hosts, then runs the lint,
versioning, and SDK jobs.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.