Problem
Cloud Hypervisor startup can pass the current guest-network readiness gate while eth0 is still down. AWF then runs the service-connectivity probe once, fails with Cloud Hypervisor guest connectivity probe failed, tears down the microVM, and exits before the agent starts.
This continues after v0.28.2 and the fixes in #7555 and #7568. Fleet evidence is tracked in github/gh-aw#54402: 11 of 16 failures in the sampled window had the same signature, including lo reported UP while eth0 remained state DOWN.
The current gap is in CloudHypervisorRuntimeBackend:
waitForGuestNetworkReady() checks only ip link show dev lo.probeGuestConnectivity() immediately performs a one-shot Squid connectivity check.- A transient
eth0 bring-up delay is therefore treated as a fatal infrastructure failure.
- The caller has no structured way to distinguish a retryable pre-agent boot failure from a permanent startup or agent-command failure.
The compiler-side retry proposed in github/gh-aw#54405 will be closed, so AWF must own the complete recovery path rather than relying on callers to parse logs and rerun the entire command.
Required behavior
AWF must not run the wrapped agent command until the Cloud Hypervisor guest data plane and all required trusted-service paths are ready.
1. Complete guest data-plane readiness
Extend the readiness gate beyond loopback. Before service probes begin, verify with bounded backoff that:
lo is UP and has 127.0.0.1/8.- The configured guest interface (
eth0) is UP/operational.
eth0 has the expected guest address.- The expected route to the trusted-service network is installed.
Readiness checks should use the generated network plan rather than hard-coded values where possible.
2. Retry transient connectivity failures in the same microVM
After interface/address/route readiness succeeds, retry transient Squid, API proxy, and topology-peer connectivity failures with bounded backoff inside the existing microVM boot. Preserve per-leg diagnostics so the exhausted error identifies which service remained unavailable.
Permanent configuration, policy, authentication, and topology errors must remain fail-fast rather than being hidden by broad retries.
3. AWF-owned bounded boot recovery
If the guest data plane or trusted-service connectivity still fails after the phase-level readiness window, AWF should perform a bounded Cloud Hypervisor boot retry when—and only when—the failure is classified as retryable and the wrapped agent command has not started.
Equivalent reliability to the retired compiler mitigation is required:
- Maximum 3 total boot attempts.
- Linear backoff of 5 seconds then 10 seconds, unless measurements justify a better bounded policy.
- Cloud Hypervisor only; other runtimes remain unchanged.
- Recreate and clean up each failed partial VM through the normal manager lifecycle.
- Preserve diagnostics for every failed attempt and label attempt counts in logs.
- Return the final AWF exit status without requiring callers to inspect human-readable log signatures.
The implementation must enforce the pre-agent invariant structurally. Do not infer safety by searching accumulated logs. Once the wrapped agent command has been dispatched, AWF must never automatically rerun it.
4. Structured failure classification
Introduce an internal typed/structured classification for retryable Cloud Hypervisor readiness failures instead of matching strings such as Cloud Hypervisor guest connectivity probe failed or guest-network-not-ready.
The final exhausted error should include:
- lifecycle stage and attempt count;
- interface state and addresses;
- routing state;
- failed connectivity leg and its stderr/exit status;
- locations of preserved per-attempt diagnostics.
Why this belongs in AWF
github/gh-aw#54405 retries the entire AWF invocation based on signatures in one append-only log. That approach can restart more than the failed lifecycle phase, can duplicate side effects if execution advanced unexpectedly, and can misclassify a later unrelated failure because an earlier retry signature remains in the log. AWF owns the microVM lifecycle and knows definitively whether agent execution has begun, so it is the only layer that can retry safely and precisely.
Regression coverage
Add deterministic unit tests for:
- delayed loopback,
eth0, address, and route readiness;
- delayed Squid, API proxy, and topology-peer reachability;
- phase-level retry exhaustion with complete diagnostics;
- successful recovery without recreating the VM when readiness converges;
- bounded VM recreation after a classified pre-agent boot failure;
- no boot retry for permanent startup failures;
- no retry after the wrapped agent command starts or fails;
- cleanup and diagnostic preservation for every failed boot attempt;
- no behavior change for non-Cloud Hypervisor runtimes.
Add a live-KVM stress case that exercises concurrent or deliberately delayed guest-network initialization so the fleet timing failure is represented, not only the fast-path boot seen in ordinary CI.
Success criteria
- Delayed
eth0 initialization and transient trusted-service reachability recover within bounded readiness windows.
- Persistent retryable boot races receive at most 3 AWF-owned boot attempts with bounded backoff.
- All retries happen before the wrapped agent command starts; the command executes at most once.
- A genuinely unavailable trusted service fails after the bound with actionable per-attempt network diagnostics.
- Callers invoke AWF once and do not need runtime-specific retry wrappers or log-signature matching.
- Existing loopback readiness, connectivity probing, cleanup, exit-code propagation, and non-Cloud Hypervisor behavior remain covered.
- Fleet validation shows no recurring
Cloud Hypervisor guest connectivity probe failed zero-turn startup cluster across the next comparable monitoring window.
References
Problem
Cloud Hypervisor startup can pass the current guest-network readiness gate while
eth0is still down. AWF then runs the service-connectivity probe once, fails withCloud Hypervisor guest connectivity probe failed, tears down the microVM, and exits before the agent starts.This continues after v0.28.2 and the fixes in #7555 and #7568. Fleet evidence is tracked in github/gh-aw#54402: 11 of 16 failures in the sampled window had the same signature, including
loreported UP whileeth0remainedstate DOWN.The current gap is in
CloudHypervisorRuntimeBackend:waitForGuestNetworkReady()checks onlyip link show dev lo.probeGuestConnectivity()immediately performs a one-shot Squid connectivity check.eth0bring-up delay is therefore treated as a fatal infrastructure failure.The compiler-side retry proposed in github/gh-aw#54405 will be closed, so AWF must own the complete recovery path rather than relying on callers to parse logs and rerun the entire command.
Required behavior
AWF must not run the wrapped agent command until the Cloud Hypervisor guest data plane and all required trusted-service paths are ready.
1. Complete guest data-plane readiness
Extend the readiness gate beyond loopback. Before service probes begin, verify with bounded backoff that:
lois UP and has127.0.0.1/8.eth0) is UP/operational.eth0has the expected guest address.Readiness checks should use the generated network plan rather than hard-coded values where possible.
2. Retry transient connectivity failures in the same microVM
After interface/address/route readiness succeeds, retry transient Squid, API proxy, and topology-peer connectivity failures with bounded backoff inside the existing microVM boot. Preserve per-leg diagnostics so the exhausted error identifies which service remained unavailable.
Permanent configuration, policy, authentication, and topology errors must remain fail-fast rather than being hidden by broad retries.
3. AWF-owned bounded boot recovery
If the guest data plane or trusted-service connectivity still fails after the phase-level readiness window, AWF should perform a bounded Cloud Hypervisor boot retry when—and only when—the failure is classified as retryable and the wrapped agent command has not started.
Equivalent reliability to the retired compiler mitigation is required:
The implementation must enforce the pre-agent invariant structurally. Do not infer safety by searching accumulated logs. Once the wrapped agent command has been dispatched, AWF must never automatically rerun it.
4. Structured failure classification
Introduce an internal typed/structured classification for retryable Cloud Hypervisor readiness failures instead of matching strings such as
Cloud Hypervisor guest connectivity probe failedorguest-network-not-ready.The final exhausted error should include:
Why this belongs in AWF
github/gh-aw#54405 retries the entire AWF invocation based on signatures in one append-only log. That approach can restart more than the failed lifecycle phase, can duplicate side effects if execution advanced unexpectedly, and can misclassify a later unrelated failure because an earlier retry signature remains in the log. AWF owns the microVM lifecycle and knows definitively whether agent execution has begun, so it is the only layer that can retry safely and precisely.
Regression coverage
Add deterministic unit tests for:
eth0, address, and route readiness;Add a live-KVM stress case that exercises concurrent or deliberately delayed guest-network initialization so the fleet timing failure is represented, not only the fast-path boot seen in ordinary CI.
Success criteria
eth0initialization and transient trusted-service reachability recover within bounded readiness windows.Cloud Hypervisor guest connectivity probe failedzero-turn startup cluster across the next comparable monitoring window.References