Skip to content

EmbeddedMavenExecutorTest hangs in CI until the job timeout #38

Description

@slachiewicz

EmbeddedMavenExecutorTest hangs in CI and never fails on its own. The job produces no further output and stays alive until something external kills it, so the run is reported as cancelled rather than red. That misfiles easily as CI flake or as a bad dependency bump, which is what happened on #37.

Signature

In a healthy run the fail-fast job Verify / ubuntu-latest jdk-17-zulu 3.10.0-rc-1 finishes in about four minutes and EmbeddedMavenExecutorTest itself takes roughly 15 seconds. When the hang occurs, the log stops immediately after

[INFO] Running org.apache.maven.executor.embedded.EmbeddedMavenExecutorTest

and emits nothing again until The operation was canceled. DockerExeExecutorTest and TestContainersExecutorTest, which normally follow, never start. The two test classes that run before it pass with normal timings, and their relative order varies between runs, so the stall is specific to EmbeddedMavenExecutorTest and not to whatever happens to run last.

Every occurrence ends with the same three orphan processes reaped during cleanup:

Terminate orphan process: pid (2495) (java)
Terminate orphan process: pid (2718) (sh)
Terminate orphan process: pid (2719) (java)

The first is the Surefire fork. The sh and java pair is a child process that outlived the test and was still running when the runner tore the job down.

Occurrences

Run Job Test started Killed Stalled for Ended by
31342483530 93318442288 2026-08-09 23:42:39Z 00:04:40Z 22 min a later push
31343457497 93396388296 2026-08-10 08:26:38Z 14:25:31Z 5 h 59 min timeout-minutes: 360
32004620308 95311702729 2026-08-17 07:13:17Z 07:26:30Z 13 min ff-timeout-minutes: 15
30933906017 95830701947 2026-08-18 18:55:55Z 19:11:11Z 15 min ff-timeout-minutes: 15

The first two ran under an older revision of apache/maven-gh-actions-shared@v5 that passed timeout-minutes: 360, so one of them burned very nearly six hours of runner time on a stalled test. The shared workflow now caps the fail-fast job at 15 minutes, which bounds the waste but still surfaces the result as cancelled.

Frequency

Across the 42 runs of that fail-fast job since 2026-06-01: 24 green with a median duration of 226 s and a range of 193 s to 285 s, 13 red for unrelated reasons, and 5 cancelled. Four of those five cancellations are this hang. The fifth is a genuine concurrency cancellation at 92 s, well before the test phase.

The hangs are not spread evenly. All four fall on or after 2026-08-09, and they account for every fail-fast job that has reached the test phase since that date. On current evidence this is not an occasional flake but a reproducible stall on ubuntu-latest, and main has had no green fail-fast job since 2026-08-04.

Why nothing catches it

  • MavenExecutorTestSupport carries @Timeout(60), but the default ThreadMode.INFERRED resolves to SAME_THREAD. In that mode JUnit measures elapsed time after the test method returns, so a method that never returns is never interrupted and the timeout never fires. There is no junit-platform.properties overriding the default thread mode.
  • Surefire has no forkedProcessTimeoutInSeconds, so the stalled fork is not killed either.
  • redirectTestOutputToFile is true in maven-executor/pom.xml, so whatever the test printed before stalling went to target/surefire-reports, which is discarded when the job is killed. That is why the console shows nothing at all.

Suggested next steps

  • Set threadMode = ThreadMode.SEPARATE_THREAD on the @Timeout in MavenExecutorTestSupport, or set junit.jupiter.execution.timeout.thread.mode.default = SEPARATE_THREAD in a junit-platform.properties. Either turns the hang into a normal test failure with a stack trace pointing at the blocked call.
  • Add forkedProcessTimeoutInSeconds to the Surefire configuration as a backstop, so a wedged fork fails the build instead of the job.
  • Upload target/surefire-reports as a workflow artifact when the build does not succeed, so the per-test output survives a kill and the stalling method can be named.
  • The orphaned sh and java pair is the most promising lead on the root cause. Identifying which of the 15 inherited test methods leaves a child process behind would likely explain the stall directly.

This issue was created with AI assistance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingbuildPull requests that change the build processpriority:blockerBlocks development and/or testing work, production could not run

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions