Test native-image behavior in CI: nothing ever runs the built binary

[rochala]

Problem

#131 fixed a bug where no Scala 3 artifact had ever produced a docstring from a released native binary. The entire test suite passed throughout, and it would still pass if the bug were reintroduced.

Two distinct gaps let that happen.

1. No test asserts docstrings appear at all

grep -r docstring lib/test/src cli/test/src returns nothing. DocstringExtractor has no direct coverage and no integration-level assertion that a docstring reaches stdout.

This is cheap to close, and the fixtures are already set up for it — e.g. fixtureScala3/src/cellar/fixture/scala3/CellarTrait.scala carries /** Sealed ADT hierarchy for testing sealedChildren extraction. */, so an IntegrationTest case can fetch it from the local m2 repo and assert the text appears.

Note this alone would not have caught #131: the JVM path worked correctly. It closes a real coverage hole, not this bug class.

2. Nothing ever runs the native binary

This is the gap that matters. The failure mode of #131 was JVM works, native-image silently differs — invisible to ./mill _.test, which only ever runs on the JVM.

ci.yml runs ./mill _.test and nix flake show. cli.nativeImage is built only in snapshot.yml and release.yml, and in both cases the binary is packaged and published without being executed once. A binary that cannot resolve a stdlib, or cannot start at all, ships green.

Proposal

Add a native smoke job to ci.yml: build cli.nativeImage, then run the resulting binary and assert on real output. At minimum:

• --version exits 0 and reports native-image as the runtime (CellarApp.scala:18 already distinguishes it)
• get-external on a Scala 3 artifact prints a docstring — the Fix missing docstrings in the native-image binary #131 regression, and the one assertion that requires the coursier stdlib fetch to have resolved a real version
• get-external on a Java artifact prints a signature — exercises the bundled-JRE path (bundledJreJar, build.mill:111), which is native-only by construction and equally untested

Scope and cost notes:

• One target (linux-x86_64) is enough; the divergences are runtime/resource-embedding issues, not per-arch. No need to mirror snapshot.yml's 4-way matrix.
• The native build takes ~90s locally warm, longer cold in CI once the GraalVM JVM download is counted. Put it in its own job with its own timeout-minutes rather than inside the existing test job, which is capped at 15.
• Prefer the locally published fixtures (./mill publishFixtures) over a third-party coordinate for the docstring assertion, so CI does not depend on a Maven Central artifact staying put. The stdlib fetch still hits the network either way, which is the point.

Other native-only surfaces worth asserting once the harness exists

These share the #131 failure mode — no JVM test can see them:

• resource embedding generally (this bug was compiler.properties not being embedded while library.properties was)
• reflection/proxy config reachable only at image runtime
• --java-home fallback and JRT filesystem behavior (graal#10013 is already worked around by hand)
• slf4j-nop / logging init under native-image

Acceptance

• CI fails if a native binary is built that cannot print a Scala 3 docstring.
• Reverting Fix missing docstrings in the native-image binary #131's one-line change makes that job go red.

🤖 Generated with Claude Code

[rochala]

The smoke job must run the published artifact on a fresh runner, not a binary it just built

New evidence for this issue, from the M12 release candidate (run 32705875933): cellar get-external org.typelevel:cats-effect_3:3.6.1 cats.effect.IO printed a dotc crash report into the middle of its own Markdown output.

Root cause: -H:+AllowJRTFileSystem puts the jrt provider in the image, and jdk.internal.jrtfs.SystemImage is initialized at image-build time — so the binary carries the build machine's java.home baked in. Opening jrt:/ anywhere else throws InternalError: NoSuchFileException /home/runner/.cache/coursier/.../lib/modules, which dotc's JrtClassPath does not recover from (it only catches ProviderNotFoundException/FileSystemNotFoundException), so the docstring inspector died and Driver.doCompile printlned the crash over stdout.

The part that matters for this ticket: the proposal as written would not have caught it. A job that builds cli.nativeImage and then runs the binary on the same runner still has the build JDK sitting at the baked path, so jrt:/ resolves and the docstring comes out fine. That is exactly why #131 was "verified by rebuilding the native image" locally and still shipped broken — a locally built binary is verified on the one machine where the bug is invisible.

So the acceptance criterion needs a second half:

• Build and run must not share a filesystem. Download the packaged .tar.gz from the build job's artifacts in a separate job (fresh runner), unpack, and run the assertions there. Same for release.yml: gate publication on a job that consumes the uploaded artifact, not on the job that produced it.
• Run at least one assertion with no JDK visible — env -u JAVA_HOME, and ideally on a runner/container without a JDK installed at all. Anything the binary can reach through an ambient JDK is a path that will differ on a user's machine.

This bug class — works on the machine that built it, fails everywhere else — is a superset of the #131 class and is invisible to any same-runner check. Worth adding to the "other native-only surfaces" list, since it applies to anything baked at image-build time (paths, embedded resources, statics of JDK-internal classes).

Reproduction to keep as a regression assertion, since it is cheap and catches both classes:

env -u JAVA_HOME ./cellar get-external org.typelevel:cats-effect_3:3.6.1 cats.effect.IO
# must exit 0, print the IO docstring, and stdout must contain no "Exception while compiling"

Verified against the M12 artifact (fails) and against a patched build with the local GraalVM moved aside (passes).

[rochala]

Concrete shape: a smoke job in release.yml, gating publication

Following up on the previous comment — the "run it on a machine that did not build it" part fits release.yml as one new job between build and publish-maven, no changes to the existing jobs beyond one needs: line.

  smoke:
    name: Smoke (${{ matrix.target }})
    needs: build
    strategy:
      fail-fast: false
      matrix:
        include:
          - { os: ubuntu-latest,    target: linux-x86_64 }
          - { os: ubuntu-24.04-arm, target: linux-aarch64 }
          - { os: macos-latest,     target: macos-arm64 }
          - { os: macOS-15-intel,   target: macos-x86_64 }
    runs-on: ${{ matrix.os }}
    permissions:
      contents: read
    steps:
      - name: Download packaged binary
        uses: actions/download-artifact@v8
        with:
          name: ${{ matrix.target }}
      - name: Unpack
        run: mkdir -p staging && tar -xzf cellar-${{ inputs.version }}-${{ matrix.target }}.tar.gz -C staging
      - name: Run the binary
        shell: bash
        env:
          JAVA_HOME: ''
        run: |
          set -euo pipefail
          CELLAR="$PWD/staging/cellar"

          "$CELLAR" --version | tee version.txt
          grep -q 'native-image' version.txt

          "$CELLAR" get-external org.typelevel:cats-effect_3:3.6.1 cats.effect.IO > io.md
          grep -q 'A pure abstraction representing the intention' io.md
          ! grep -q 'Exception while compiling' io.md

          "$CELLAR" get-external com.google.guava:guava:33.4.0-jre com.google.common.collect.ImmutableSet > guava.md
          grep -q 'class ImmutableSet' guava.md

and then:

  publish-maven:
    needs: [build, jar, smoke]

Notes on the details that matter:

• No actions/checkout. The job must touch nothing from the repo — only the packaged artifact, exactly as a user would get it.
• The three assertions cover three distinct native-only failure modes. --version catches "cannot start at all"; the Scala 3 docstring catches both the Fix missing docstrings in the native-image binary #131 stdlib-fetch bug and the M12 jrt bug; the Exception while compiling check catches compiler diagnostics leaking into the Markdown even when the query otherwise looks fine (the M12 symptom — output was present, just corrupted). Guava exercises the bundled JRE path.
• JAVA_HOME: '' so nothing resolves through an ambient JDK. The runners still have a JDK on PATH; a JDK-less container would be stricter, but nothing in the native path consults PATH.
• Gate publish-maven, not just release. A Sonatype release is immutable, so the binaries have to be proven runnable before that step, not before the GitHub Release.

Verified the gate discriminates, running that script verbatim against both binaries with JAVA_HOME='':

M12 released artifact  → exit 1   (red)
patched build          → exit 0   (green)

Cost is one extra runner per target for a few seconds each (no build — it only downloads and runs), plus the network fetches the assertions make.