ci: keep build/venv when the build retry nukes the build directory - #1814
ci: keep build/venv when the build retry nukes the build directory#1814sbryngelson wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Updates the CI build retry logic to preserve build/venv across retries, preventing Frontier compute-node jobs from failing due to inability to reach PyPI after the build directory is wiped.
Changes:
- Add
nuke_build()helper to delete everything underbuild/exceptvenv. - Replace
rm -rf buildwithnuke_buildin both retry paths. - Expand script header comment with Frontier/PyPI context tied to #1813.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| : "${MFC_BUILD_RETRY_DELAY:=30}" | ||
|
|
||
| nuke_build() { | ||
| find build -mindepth 1 -maxdepth 1 ! -name venv -exec rm -rf {} + 2>/dev/null || true |
| @@ -24,7 +30,7 @@ retry_build() { | |||
| echo "Post-build validation failed on attempt $attempt." | |||
| if [ $attempt -lt $max_attempts ]; then | |||
| echo " Nuking build directory before retry..." | |||
There was a problem hiding this comment.
Reworded in 608e137: the log now says the build directory is cleared with build/venv kept.
| @@ -39,7 +45,7 @@ retry_build() { | |||
| fi | |||
| if [ $attempt -lt $max_attempts ]; then | |||
| echo " Build failed — nuking build directory before retry..." | |||
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1814 +/- ##
=======================================
Coverage 62.26% 62.26%
=======================================
Files 84 84
Lines 21558 21558
Branches 3188 3195 +7
=======================================
Hits 13423 13423
Misses 5937 5937
Partials 2198 2198 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Folded into #1800 ( One thing changed in the process: #1800 removes the cluster-wide outage breaker entirely, so the sentence here about a genuine PyPI failure still marking it no longer holds. The reason for keeping Closing in favour of #1800; reopen if you'd rather land this separately. |
Fixes #1813.
retry_buildanswered a failed first attempt withrm -rf build, which also removedbuild/venv. On Frontier the venv is fetched on the login node because compute nodes have no route to PyPI, so attempt 2 could never reinstall it, andclassify-build-failure.shread the resultingFailed to fetch https://pypi.org/...as a cluster-wide outage. One compile error on one branch then exited every Frontier job on every PR with code 78 for twenty minutes, and refreshed the marker each time it happened again. Today it was tripped twice (18:01 by a syscheck install timeout on another branch, 19:14 by a compile error on a probe branch) and skipped the Frontier lanes on #1805, #1807 and #1811.The retry now removes everything under
build/exceptvenv. Checked in a scratch tree: after two failed attempts onlybuild/venvremains. Nothing else changes; a genuine PyPI failure on the login node still marks the breaker.