Skip to content

fix(ci): bound package-test child commands so a hang fails fast - #582

Open
goosewobbler wants to merge 1 commit into
mainfrom
fix/bound-package-test-commands
Open

fix(ci): bound package-test child commands so a hang fails fast#582
goosewobbler wants to merge 1 commit into
mainfrom
fix/bound-package-test-commands

Conversation

@goosewobbler

Copy link
Copy Markdown
Contributor

A wedged child process in scripts/test-package.ts had nothing to stop it. No command carried a timeout, and the package job set no timeout-minutes, so it inherited GitHub's 6-hour default.

That is not hypothetical — it cost ~2 hours of a macOS runner on #580 before I cancelled it manually:

13:11:05  Installing local packages for electron-forge-app-cjs...
13:11:17  [WARN] GET .../@esbuild/aix-ppc64... error (ENOTCONN). Will retry in 10s
13:11:22  Done in 16s using pnpm v11.20.0
15:03:43  ##[error]The operation was canceled.        <- manual, ~114 min in

pnpm printed its completion summary and then never exited, after the registry connection dropped mid-install (~40 ENOTCONN retries immediately prior). execSync waited on a child that was never coming back. The step normally takes seconds.

Nothing here is specific to that failure mode — any child that stops making progress produces the same outcome.

Changes

  • Per-command bound, 30 minutes. Applied in both execCommand and execCommandAsync, so the browser-mode path is covered too. Overridable via PACKAGE_TEST_COMMAND_TIMEOUT_MS for a cold Rust build on a slow machine.
  • A timeout is terminal, not retried. The existing Windows retry exists for pnpm exiting non-zero without explanation; retrying a hang just spends the budget twice. Detected by elapsed time rather than by signal, so an unrelated SIGKILL still takes the normal retry path.
  • Job backstop, 45 minutes, matching the convention already used across this repo (_ci-lint 10, _ci-smoke-autoinstall-driver 15, _ci-rust-clippy 30, the mobile builds 40–60). Catches anything the per-command bound can't see.

Calibration

Measured across 42 Package - * jobs on the last three main runs: slowest is 15.1 minutes end to end (Package - Electron [Windows] - esm), and that whole job is many commands. No single command comes close to 30 minutes, so the bound only trips on something genuinely stuck — while cutting the worst case from 6 hours to 45 minutes.

Verification

  • Timeout path exercised directly: execSync with a 2s bound on sleep 30 is killed at 2002ms and takes the timeout branch, not the retry branch.
  • node ./scripts/test-package.ts --package=electron-script-app-esm passes end to end on this branch.
  • format:check and lint clean.

Found while investigating #580, kept separate since it's unrelated to that upgrade and applies equally on main today.

A wedged child in test-package.ts had nothing to stop it: no command carried a
timeout and the package job set no timeout-minutes, so it inherited GitHub's
6-hour default. One run burned ~2 hours of a macOS runner after pnpm printed
its "Done" summary and then never exited, the registry connection having
dropped mid-install.

Bound each command at 30 minutes (env-overridable for a cold Rust build) and
add a 45-minute job backstop, against a slowest-real-job time of ~15 minutes.
A timeout is terminal rather than retried — retrying a hang just spends the
budget twice — and reports which command stalled instead of an opaque kill.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CsgeivZCqduTQmWK55zqiT
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor
Release Preview — no release

No bump label detected.
Note: Add bump:patch, bump:minor, or bump:major to trigger a release.


Updated automatically by ReleaseKit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant