Crash handler core dedup - #16
Closed
nchaimov wants to merge 80 commits into
Closed
Conversation
Use --external-launcher for Slurm launch; fallback to old behavior; add srun wrapper CI job
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@4d04d5d...d7f5e7f) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 4.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [docker/setup-compose-action](https://github.com/docker/setup-compose-action) from 2.1.0 to 2.2.0. - [Release notes](https://github.com/docker/setup-compose-action/releases) - [Commits](docker/setup-compose-action@8cccb8c...16feee7) --- updated-dependencies: - dependency-name: docker/setup-compose-action dependency-version: 2.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 7.1.0 to 7.2.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@bcafcac...f9f3042) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: 7.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.1.0 to 4.2.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@4907a6d...650006c) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 4.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…ithub/workflows/docker/login-action-4.2.0 Bump docker/login-action from 4.1.0 to 4.2.0 in /.github/workflows
…ithub/workflows/docker/build-push-action-7.2.0 Bump docker/build-push-action from 7.1.0 to 7.2.0 in /.github/workflows
…ithub/workflows/docker/setup-compose-action-2.2.0 Bump docker/setup-compose-action from 2.1.0 to 2.2.0 in /.github/workflows
…ithub/workflows/docker/setup-buildx-action-4.1.0 Bump docker/setup-buildx-action from 4.0.0 to 4.1.0 in /.github/workflows
Previously, chosen_realized_cachepath was copied into set_intercept_readlink_cachepath() chosen_realized_cachepath and chosen_parsed_cachepath were copied into set_should_intercept_cachepath() This PR removes both setter functions and makes the original pointers global.
Removes chosen_cachepath and cachepath_bitindex from spindle_launch.h Updates initialization of matching variables in ldcs_process_data. determineValidCachePaths() moved from spindle_be.cc to ldcs_audit_server_process.c to get ldcs_process_data visibility. Added #include "parseloc.h" to ldcs_audit_server_process.c to get declaration of determineValidCachePaths(). Relocated "parseloc.h" to src/util so ldcs_audit_server_process.c could find it. Trued up signedness of types caused my making "parseloc.h" more visible, e.g., cachepath_bitidx is now uint64_t everywhere.
The three-message-reply response is now a single message with two strings. The symbolic version of the cachepath is no longer communicated as it was not being used.
New name is ldcs_audit_server_md_allreduce_AND(). If we get to the point where we're using other allreduce operations we can solve the problem of duplicating the op list in md-land and cobo-land. For now, we're only using one op in md-land, so the op can go into the function name.
That configure parameter is no longer supported.
Replaced with
--with-cachepaths=/tmp/commpath/cachepath
--with-commpath=/tmp/commpath
Replaced assert() with return -1 in:
src/client/beboot/spindle_bootstrap.c
src/client/client/client.c
Removed assert() with no replacement in:
src/client/client/client.c
src/client/client/intercept_readlink.c
src/client/client/should_intercept.c
Created Issue llnl#187 to remove debugging code in send_cachepath_query()
send_cachepath_query() now has delay_between_retries of 0.1 seconds and max 1000 retries.
Also returns immediately in case of network errors.
Also uses spindle_strdup() instead of strdup().
src/utils/parseloc.c
src/server/auditserver/ldcs_audit_server_handlers.c
Removed/reclassified logging statements.
Fix issue with 'bin' being generated in random file names and causing test failures
Cleans up three warnings via -W[all|extra|error].
Adds server-side components of crash handling. Client sends LDCS_MSG_CRASH_REPORT to sever. Server propagates request. When a single rank is chosen for a given crashsite, replies with LDCS_MSG_CRASH_RESPONSE.
Add the client-side crash handler. On a fatal signal, it chains to an application-registered signal handler, if any, and, if the application handler did not fix the fault, sends LDCS_MSG_CRASH_REPORT to the server which selects a single process per crashsite. Non-selected crashsites set their coredump limit to zero.
Adds --crash-dedup option to control whether crash deduplication is enabled.
Adds --enable-crash-handler to control whether the crash handler is compiled. Adds --enable-crash-dedup to control whether crash deduplication is enabled by default at runtime.
Adds a new script run_crash_tests_template.sh which runs various configurations of the new crash_test.c testsuite. This tests various kinds of crashes, numberes of distinct crashsites, etc. to validate that the crash handler deduplicates them correctly. It also tests chaining to application signal handlers that use the safepoint pattern to fix faults and retry.
Run the new crash handler testsuite in CI. Configure CI runner to generate coredumps, add shared filesystem to aggregate coredumps across nodes in mutli-node tests. Grants CAP_SYS_RESOURCE so ulimit can be set in containers.
autotools-generated files re-generated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test PR to force CI to run