Skip to content

Crash handler logging, documentation - #197

Open
nchaimov wants to merge 8 commits into
llnl:develfrom
ParaToolsInc:crash-handler-logs
Open

Crash handler logging, documentation#197
nchaimov wants to merge 8 commits into
llnl:develfrom
ParaToolsInc:crash-handler-logs

Conversation

@nchaimov

@nchaimov nchaimov commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

This PR extends the crash handler:

  • The alternate stack to handle crashes due to stack overflow is now optional and off by default. It is enabled with --crash-altstack. In the event of a stack overflow without an alternate stack, the kernel will restore the default signal handler automatically and core dumps will not be deduplicated.
  • The crash handler now optionally produces a log showing which ranks crashed at which sites. This is enabled with --crash-log=[path]. If the optional path is not provided, the log will be written to spindle-crash-log.$NUMBER in the working directory; if it is provided, it will be written at the provided path.
  • To enable use with the Slurm plugin, this PR also adds the ability to specify Spindle arguments when starting a session through salloc and sbatch; this functionality was previously missing.
  • The Flux plugin now handles the new arguments.
  • A new documentation file doc/spindle_crash_handler_README.md describes the use of the crash handler.
  • The manpage is updated with the new arguments.
  • The launcher documentation doc/spindle_launch_README.md is updated with the new options.
  • The crash handler tests now include session mode tests and validation of crash log files.

The crash handler logging is implemented as follows: previously, as soon as a server recognized a crash site it had previously seen, it immediately sent a not-selected reply and did not pass the message further up the tree. Now, if crash logging is enabled, the not-selected reply is still sent immediately, but an additional message is passed up the tree to report an additional crashing rank. Crash reports also now include a display rank representing the MPI or scheduler-provided rank as determined from environment variables. The root server maintains a table of crash sites and associated display ranks. When the root server exits, it produces a log file having the format:

`exe: /home/me/my_app
site: libsolver.so.1+0x2f10
exemplar: 4
count: 13
ranks: 4-15,18

exe: /home/me/my_app
site: abort:my_app: solver.c:88: solve: Assertion n > 0' failed. exemplar: 0 count: 4 ranks: 0-3

Each entry lists the executable that crashed, the library+offset or assert message associated with the failure, the rank of the exemplar core dump that was selected to be output, the total number of ranks that crashed at that crash site, and a comma-separated list of ranges of ranks that crashed at that site.

Because the log is written when the server exits, when session mode is used, a single log is produced for the whole session, with coredumps deduplicated across the whole session. To handle multiple executables being run within a session, the executable is now always included in the crash key so that crashes at the same site in a given library called from two different executables will not be deduplicated.

Two new tests are added to verify behavior in session mode: one test in which the same executable crashes in the same site in two different jobs within a session, and another test in which two different executables crash at the same site in a library. The crash tests now verify that the expected crash sites and ranks are recorded in the log file and match the crash site in the core dump.

nchaimov added 8 commits July 30, 2026 19:17
--crash-log enables logging of which ranks crashed at which callsites. --crash-log takes optional path to log location
When run in session mode, the log can span multiple jobs. Always include executable in crashsite key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant