fix: recover transient Cloud Hypervisor readiness failures - #7611
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d69f30be-77fd-415b-aa99-5f050637b2e9
lpcox
changed the title
lpcox
changed the title
|
| Metric | Base | PR | Delta |
|---|---|---|---|
| Lines | 93.67% | 93.67% | ➡️ +0.00% |
| Statements | 92.53% | 92.53% | ➡️ +0.00% |
| Functions | 93.05% | 92.77% | 📉 -0.28% |
| Branches | 85.80% | 85.79% | ➡️ -0.01% |
📁 Per-file Coverage Changes (3 files)
| File | Lines (Before → After) | Statements (Before → After) |
|---|---|---|
src/cloud-hypervisor/manager.ts |
90.7% → 89.5% (-1.19%) | 89.3% → 88.2% (-1.13%) |
src/cloud-hypervisor-runtime-backend.ts |
96.9% → 96.6% (-0.28%) | 94.6% → 94.2% (-0.32%) |
src/log-directory-setup.ts |
96.2% → 100.0% (+3.78%) | 96.3% → 100.0% (+3.71%) |
Coverage comparison generated by scripts/ci/compare-coverage.ts
There was a problem hiding this comment.
Pull request overview
Adds bounded Cloud Hypervisor recovery for transient guest-network and trusted-service readiness failures.
Changes:
- Verifies complete guest network readiness before connectivity probes.
- Retries connectivity legs and recreates failed VMs up to three attempts.
- Preserves per-attempt diagnostics and expands regression tests/documentation.
Show a summary per file
| File | Description |
|---|---|
src/cloud-hypervisor/manager.ts |
Exposes guest network-plan properties. |
src/cloud-hypervisor-runtime-backend.ts |
Implements readiness checks, retries, recovery, and diagnostics. |
src/cloud-hypervisor-runtime-backend.test.ts |
Tests recovery and failure classification. |
docs/cloud-hypervisor-foundation.md |
Documents readiness and recovery behavior. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 4/4 changed files
- Comments generated: 5
- Review effort level: Balanced
| name: 'api-proxy', | ||
| command: | ||
| `unset HTTP_PROXY HTTPS_PROXY http_proxy https_proxy ALL_PROXY all_proxy; ` + | ||
| `wget -q -T ${CLOUD_HYPERVISOR_API_PROXY_PROBE_TIMEOUT_SECONDS} ` + |
| } catch (error) { | ||
| throw new CloudHypervisorRetryableReadinessError( | ||
| 'guest-network-readiness', | ||
| bootAttempt, | ||
| `guest-network-not-ready: expected lo UP with 127.0.0.1/8, ` + | ||
| `${guestInterfaceName} state UP with ${expectedAddress}, and default route via ` + | ||
| `${guestGatewayIp} (${formatError(error)})`, | ||
| error, |
| const failure = | ||
| `Cloud Hypervisor guest connectivity probe failed with exit code ${result.exitCode}` + | ||
| (detail ? ` (${detail})` : ''), | ||
| (detail ? ` (${detail})` : ''); | ||
| if (result.exitCode === 126 || result.exitCode === 127) { | ||
| throw new Error(`Cloud Hypervisor guest connectivity configuration is invalid: ${failure}`); | ||
| } | ||
| throw new CloudHypervisorRetryableReadinessError( |
| private getBootDiagnosticsDirectory(bootAttempt: number): string { | ||
| const root = this.config.auditDir | ||
| ? `${this.config.auditDir}/cloud-hypervisor` | ||
| : `${this.config.workDir}/diagnostics/cloud-hypervisor`; | ||
| return `${root}/boot-attempt-${bootAttempt}`; |
| }); | ||
|
|
||
| it('waits with bounded backoff for guest loopback before probing connectivity', async () => { | ||
| it('waits with bounded backoff for the complete guest data plane before probing connectivity', async () => { |
|
@copilot address review feedback |
|
⏳ Copilot review left inline comments. @copilot To proceed:
|
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤
|
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.
|
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓
|
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓
|
|
🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "example.com"See Network Configuration for more information.
|
|
❌ Smoke Gemini reports failed. Facets need polishing...
|
|
✅ Smoke Claude passed
|
|
🔌 Smoke Services — All services reachable! ✅
|
|
✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓
|
|
❌ Contribution Check failed. Please review the logs for details.
|
|
📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤
|
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 Warning Firewall blocked 3 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "msfeed2.pkgs.visualstudio.com"
- "msfeed25.pkgs.visualstudio.com"
- "registry.npmjs.org"See Network Configuration for more information.
|
|
🚀 Security Guard has started processing this pull request |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅
|
Smoke Test: Copilot BYOK (Direct Mode) ✅Status: PASS
Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY) via api-proxy sidecar
|
|
Smoke Test: Copilot Engine
Overall: PASS cc
|
Smoke Test: Claude Engine Validation
Overall result: PASS
|
|
EGRESS_RESULT allow=pass deny=pass ✅ Allowed domain (github.com) reachable: Overall: PASS cc Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "example.com"See Network Configuration for more information.
|
github-actions
Bot
added
the
smoke-copilot-network-isolation
|
Smoke Test: GitHub Actions Services Connectivity
Overall: FAIL —
|
|
Docker Sbx Smoke Test —
Overall: PASS
|
|
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) Overall: PASS
|
Chroot Version Comparison Results
Overall: FAILED — Node.js version mismatch between host and chroot environment.
|
|
Smoke test: Warning Firewall blocked 3 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "msfeed2.pkgs.visualstudio.com"
- "msfeed25.pkgs.visualstudio.com"
- "registry.npmjs.org"See Network Configuration for more information.
|
|
test: fix config writer tests on macOS ✅ MCP connectivity Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra Overall: PASS
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS Note: Java Maven required a writable All ecosystems passed — adding
|
Smoke Test: API Proxy OpenTelemetry Tracing — Results
Overall: All 5 scenarios pass.
|
Summary
eth0, the planned guest address, and the expected default route before trusted-service probesValidation
npm run buildnpm run lintnpm test -- --runInBand --silentFixes #7610