llvm_passes: lower 8 and 16 bit atomics onto their containing 32 bit word - #1512
Merged
Conversation
Clang lowers __hip_atomic_load / store / exchange / fetch_* / compare_exchange_strong on char and short to load atomic i8, store atomic i16, atomicrmw i8 and cmpxchg i16, and chipStar hands those to SPIR-V as OpAtomicLoad and friends on OpTypeInt 8 / 16. OpenCL SPIR-V consumers only implement 32 and 64 bit atomics: IGC fails the module build with error: undefined reference to `_Z18__spirv_AtomicLoadPU3AS4cii' error: backend compiler failed build. and the Intel CPU OpenCL runtime with JIT session error: Symbols not found: [ _Z20atomic_load_explicitPU3AS1VU7_Atomicc12memory_order12memory_scope, ... ] which takes every kernel in the module down with it (Kokkos TestAtomicOperations on signed char, #1497). TestSubwordAtomics.hip drives every 8 and 16 bit atomic form from all four byte lanes of a word concurrently and checks the counts, so a lowering that touches the neighbouring lanes non-atomically is caught as lost updates rather than passing quietly. subwordAtomics/subword-atomics.ll runs the post-link pass pipeline over every i8 / i16 atomic instruction in the global, local and generic address spaces and fails while any of them survives. See #1497
…word
Clang lowers __hip_atomic_load / store / exchange / fetch_* /
compare_exchange on char and short to load atomic i8, store atomic i16,
atomicrmw i8 and cmpxchg i16, and both SPIR-V producers pass those through
as OpAtomicLoad, OpAtomicStore, OpAtomicIAdd, OpAtomicCompareExchange ...
on OpTypeInt 8 / 16. OpenCL SPIR-V consumers only implement 32 and 64 bit
atomics, so the module build fails inside the driver, past spirv-val:
IGC with
error: undefined reference to `_Z18__spirv_AtomicLoadPU3AS4cii'
error: backend compiler failed build.
and the Intel CPU runtime with
JIT session error: Symbols not found:
[ _Z20atomic_load_explicitPU3AS1VU7_Atomicc12memory_order12memory_scope, ... ]
Either failure takes every kernel in the module down with it, which is
how Kokkos' TestAtomicOperations on signed char sinks the whole
Kokkos_CoreUnitTest_HIP binary on Aurora.
Add HipLowerSubwordAtomicsPass, which does what LLVM's AtomicExpand does
for targets without narrow atomics: locate the aligned 32 bit word that
holds the value and the lane's bit offset in it, then
load -> 32 bit atomic load, shift, truncate
store -> cmpxchg loop replacing only the lane
atomicrmw -> cmpxchg loop applying the operation to the lane only
(every integer op, plus fadd / fsub / fmin / fmax on
16 bit floats)
cmpxchg -> 32 bit cmpxchg with the expected and new values masked
into the current word, retried while only the other
lanes changed (strong) or not at all (weak)
Ordering and syncscope are carried over unchanged; the loops' initial
read is a monotonic atomic load in the same scope. The word address is a
GEP off the original pointer rather than an inttoptr so the address space
survives and InferAddressSpaces can still narrow a generic pointer. The
pass runs after HipLowerFPAtomicMinMax so the i16 cmpxchg that one emits
for half gets lowered as well. Values are assumed naturally aligned,
which is all clang emits; an under-aligned 16 bit atomic could straddle
two words and is left in place with a warning.
Fixes #1497
Collaborator
Author
|
/run-aurora-ci |
Collaborator
Author
|
/run-aurora-ci |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clang lowers
__hip_atomic_*oncharandshortto i8 and i16 atomic instructions, and both SPIR-V producers pass them through asOpAtomicLoad,OpAtomicIAdd,OpAtomicCompareExchangeand friends on 8 and 16 bit integers, which OpenCL SPIR-V consumers do not implement: IGC fails the module build withundefined reference to _Z18__spirv_AtomicLoadPU3AS4ciiand the Intel CPU OpenCL runtime withJIT session error: Symbols not found, taking every kernel in the module down with it. This addsHipLowerSubwordAtomicsPass, which rewrites each 8 and 16 bit atomic load, store, atomicrmw and cmpxchg onto the aligned 32 bit word that contains it (atomic load plus shift for loads, cmpxchg loops touching only the affected lane for the rest), keeping the ordering and syncscope. Found with KokkosTestAtomicOperationsonsigned charon Aurora PVC.TestSubwordAtomicsdrives every form from all four byte lanes of a word concurrently, andsubwordAtomics/subword-atomics.llchecks that no i8 or i16 atomic survives the post-link pipeline.Fixes #1497