[aw-failures] [aw] Failure Investigator Report — 2026-08-13 19:10 UTC (6h)

[github-actions]

Executive summary

• Fix the Crush CLI sandbox PATH gap in Daily Code Metrics — deterministic ENOENT on Execute Crush CLI, now 2 of the last 2 daily runs (2026-08-12, 2026-08-13), and it has zero existing tracking. New sub-issue filed below.
• Escalate [aw-failures] [P0] Failure-Investigator log capture grabs post-job cleanup tail, not the actual error #52501, don't re-file it — the log-capture-truncation bug hit 3 of the 5 failures analyzed this window (Design Decision Gate ×2, §31729681682 and §31726889123; Linter Miner, §31726936445) — same signature already tracked, still unfixed, still blinding this very investigation.
• Take no action on Smoke Gemini — single 503 RetryableQuotaError after 10 built-in retry attempts (§31732491740); transient upstream Gemini outage, isolated occurrence, no code defect.

Failure cluster table

Cluster	Severity	Workflow(s)	Runs (this window)	Tracking status
Crush CLI ENOENT in sandboxed exec	P0	Daily Code Metrics and Trend Tracking Agent	§31731856180 (2026-08-13), §31628722200 (2026-08-12)	Untracked — sub-issue filed
Log capture grabs post-job cleanup tail, not the error	P0 (recurring)	Design Decision Gate 🏗️, Linter Miner	§31729681682, §31726889123, §31726936445	Tracked — #52501 (still open, still recurring)
Gemini API 503 service-unavailable after 10 retries	P2	Smoke Gemini	§31732491740	Untracked — not actionable, isolated/transient

Evidence

Crush CLI ENOENT — install/verify succeed, execution fails (click to expand)

agentic-workflows audit on both failing runs confirms the same signature and step position:

• Both runs: Install Crush → success, Verify Crush CLI installation → success, Execute Crush CLI → failure.
• Failing run's own log: [crush-harness] spawnSync crush ENOENT, immediately after the harness logs `awf-reflect: provider=github mapped to endpoint provider=copilot baseUrl=(apiproxy/redacted) and the proxy sidecar reports all 4 listeners (openai/anthropic/copilot/gemini) started and passed key validation.
• audit-diff against the last successful run (§31419941312, 2026-08-10) shows the failed run did reach api.githubcopilot.com and api.openai.com through the proxy (2 and 3 allowed calls) — so the sandbox network path is healthy. The failure is specifically the harness's spawnSync('crush', ...) call not finding the binary at execution time.
• Conclusion: the binary is installed and verified on the runner host, but the sandboxed/chroot exec environment used by the Execute Crush CLI step does not have it on PATH (or doesn't have the install location bind-mounted). This is a harness/sandbox integration bug, not a network or install-script bug.

Log-capture truncation — pattern confirmed 3x more today (click to expand)

All 3 runs show truncated_error_logs[].tail_lines containing only post-failure git/cleanup/OTLP-teardown output (e.g. Post job cleanup., Cleaning up /tmp/gh-aw..., git config --local --unset-all) — never the actual ##[error] line from the failing step (Execute Claude Code CLI / Execute GitHub Copilot CLI). This is the exact defect #52501 already describes and proposes a fix for (anchor capture to the failing step's log offset, not the job's absolute tail). No new issue needed — this just adds 3 more confirming runs to that existing report.

Existing issue correlation

Cluster	Match	Action taken
Log-capture truncation	#52501 (high confidence — identical signature and remediation already proposed)	Left open, no duplicate filed; new run IDs noted above
Crush CLI ENOENT	No match across 22 open agentic-workflows issues	New sub-issue filed (P0)
Gemini 503	No match, and not warranted — single transient occurrence with exhausted retries	No issue filed

No open issues were found to be fixed or stale based on this window's evidence; none were closed.

Fix roadmap

• P0: Fix the Crush CLI sandbox PATH/mount gap — see sub-issue below. Blocks Daily Code Metrics 2 days running.
• P1: None new this window.
• P2: Monitor Gemini API 503 rate — no fix needed unless it recurs across multiple runs.

Sub-issues created

• P0 — Crush CLI binary unreachable inside sandboxed exec environment (ENOENT) despite passing host-level install/verify — see linked sub-issue.

References

• §31731856180 — Daily Code Metrics, today's Crush ENOENT failure
• §31628722200 — Daily Code Metrics, 2026-08-12 Crush ENOENT failure
• §31732491740 — Smoke Gemini, transient 503

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 112.9 AIC · ⌖ 38.3 AIC · ⊞ 5.3K · ◷

• expires on Aug 20, 2026, 11:21 AM UTC-08:00