Crash handler logging, documentation - #197
Open
nchaimov wants to merge 8 commits into
Open
Conversation
--crash-log enables logging of which ranks crashed at which callsites. --crash-log takes optional path to log location
When run in session mode, the log can span multiple jobs. Always include executable in crashsite key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR extends the crash handler:
--crash-altstack. In the event of a stack overflow without an alternate stack, the kernel will restore the default signal handler automatically and core dumps will not be deduplicated.--crash-log=[path]. If the optional path is not provided, the log will be written tospindle-crash-log.$NUMBERin the working directory; if it is provided, it will be written at the provided path.sallocandsbatch; this functionality was previously missing.doc/spindle_crash_handler_README.mddescribes the use of the crash handler.doc/spindle_launch_README.mdis updated with the new options.The crash handler logging is implemented as follows: previously, as soon as a server recognized a crash site it had previously seen, it immediately sent a not-selected reply and did not pass the message further up the tree. Now, if crash logging is enabled, the not-selected reply is still sent immediately, but an additional message is passed up the tree to report an additional crashing rank. Crash reports also now include a display rank representing the MPI or scheduler-provided rank as determined from environment variables. The root server maintains a table of crash sites and associated display ranks. When the root server exits, it produces a log file having the format:
`exe: /home/me/my_app
site: libsolver.so.1+0x2f10
exemplar: 4
count: 13
ranks: 4-15,18
exe: /home/me/my_app
site: abort:my_app: solver.c:88: solve: Assertion
n > 0' failed. exemplar: 0 count: 4 ranks: 0-3Each entry lists the executable that crashed, the library+offset or assert message associated with the failure, the rank of the exemplar core dump that was selected to be output, the total number of ranks that crashed at that crash site, and a comma-separated list of ranges of ranks that crashed at that site.
Because the log is written when the server exits, when session mode is used, a single log is produced for the whole session, with coredumps deduplicated across the whole session. To handle multiple executables being run within a session, the executable is now always included in the crash key so that crashes at the same site in a given library called from two different executables will not be deduplicated.
Two new tests are added to verify behavior in session mode: one test in which the same executable crashes in the same site in two different jobs within a session, and another test in which two different executables crash at the same site in a library. The crash tests now verify that the expected crash sites and ranks are recorded in the log file and match the crash site in the core dump.