Report device-side assertion messages on stderr - #1511
Open
pvelesko wants to merge 5 commits into
Open
Conversation
A failed device-side assertion prints its message through device printf, which the OpenCL and Level Zero runtimes deliver to the process' stdout. ROCm reports the same message on stderr, and gtest death tests match their expected-error regex against captured stderr only, so Kokkos death tests that abort on the device (hip_DeathTest.abort_from_device and friends) fail on chipStar with "died but not with expected error" although the process does die with SIGABRT and the message is printed. TestDeviceAbortStderr fails an assertion on the device the way Kokkos::abort does (__assert_fail(msg, "", 0, "")) and a cmake driver runs it with stdout and stderr captured separately. It passes only when the process died with SIGABRT and the assertion message is on stderr. Currently it fails: the message is on stdout, stderr is empty.
A failed device-side assertion printed its message through device printf only, and the OpenCL and Level Zero runtimes deliver device printf to the process' stdout. ROCm reports the message on stderr, and gtest death tests match their expected-error regex against captured stderr, so Kokkos death tests that abort on the device failed on chipStar with "died but not with expected error" even though the process died with SIGABRT and the message was printed. Device printf cannot reach stderr, so the message is carried to the host instead: __assert_fail (and __assert_rtn on macOS) records it, in the same format _cl_assert_fail_print prints, in a new weak device variable __chipspv_abort_msg next to the abort flag. A claim word taken with atomicCAS lets only the first work-item write, so concurrent failures do not interleave, and the copy is bounded by the buffer. When handleAbortRequest sees the abort flag set it copies the variable to the host and writes the text to stderr before aborting (or, with CHIP_HOST_IGNORES_DEVICE_ABORT, before resetting the flag; the message is reset with it so a later abort reports its own text). HipAbort erases the variable together with the abort flag when no kernel can abort, and also when the module has no __assert_fail/__assert_rtn writer, so modules that never assert do not gain a program-scope global. The device printf to stdout is kept as it was, so the message appears on both streams when both go to the same terminal.
spirv_hip.hh defines the device assert hook as __assert_rtn on macOS (where assert() expands to it) and as __assert_fail elsewhere, so the unconditional __assert_fail call did not compile on the macOS PoCL CI lane. Call the hook the header defines for the platform; the recorded message is the same on both.
With CHIP_ENABLE_DEVICE_PROGRAM_SCOPE_GLOBALS=OFF (the rusticl lane) the
globals-as-kernel-args lowering can only convert globals whose loads sit
in kernels. __assert_fail is a weak noinline device function, so the
abort flag it sets stays a program-scope global with an initializer, and
rusticl/Mesa rejects the module at spirv_to_nir ("Initializer for
CrossWorkgroup variable not yet supported"). The same happens for any
abort() reached through a non-inlined device function, with or without
the assertion message variable, so device abort is unavailable on that
configuration and the test is registered only when the option is on,
like the other program-scope-global features.
…7-device-abort-stderr # Conflicts: # tests/runtime/CMakeLists.txt
Collaborator
Author
|
/run-aurora-ci |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A failed device-side assertion prints its message through device printf, which the OpenCL and Level Zero runtimes deliver to stdout. ROCm reports it on stderr and gtest death tests match their expected-error regex against stderr only, so Kokkos death tests that abort on the device (
hip_DeathTest.abort_from_deviceand friends) failed on Aurora PVC with "died but not with expected error" although the process did die with SIGABRT.__assert_failnow also records the formatted message in a weak device variable__chipspv_abort_msgnext to the abort flag (first work-item wins via atomicCAS), andhandleAbortRequestcopies it to the host and writes it to stderr before aborting.HipAborterases the variable when no kernel can abort or when the module has no__assert_failwriter, so modules that never assert do not gain a program-scope global.TestDeviceAbortStderrruns a device assert with stdout and stderr captured separately and passes only when the process died with SIGABRT and the message is on stderr.Fixes #1487