llvm_passes: lower volatile global accesses to relaxed device-scope atomics - #1513
Open
pvelesko wants to merge 4 commits into
Open
llvm_passes: lower volatile global accesses to relaxed device-scope atomics#1513pvelesko wants to merge 4 commits into
pvelesko wants to merge 4 commits into
Conversation
… plain loads A `volatile` load or store of a 32 or 64 bit value in global memory is emitted as OpLoad / OpStore with the Volatile memory operand, which IGC serves from L1 like any other access. CUDA compiles the same source to ld.volatile / st.volatile, which the PTX ISA defines as relaxed memory operations at system scope, and Kokkos::volatile_load relies on that in the UnorderedMap insert list walk: on Aurora PVC hip.UnorderedMap_insert counts 901 keys for 900 because a losing inserter reads a stale key. TestFixVolatileLoadLoweringSPIRV.bash compiles TestFixVolatileLoadLowering.hip with --save-temps and checks the lowered device bitcode (and the SPIR-V module when the Khronos translator produced it) for relaxed atomic forms of the 32 and 64 bit global accesses, and that 16 bit and work-group local ones stay as they are. TestFixKokkosUnorderedMapInsert.hip is the insert idiom itself (claim, publish with __threadfence, relaxed CAS, volatile list walk) checked on the host for duplicate keys and orphaned slots; it passes on the CPU OpenCL runtime before and after and is meant for PVC.
…tomics
CUDA compiles a volatile access to global memory into PTX ld.volatile /
st.volatile, which the PTX ISA (8.4.2 "volatile Operation") defines as
"equivalent to a relaxed memory operation with system-scope": a strong
access that bypasses the core's L1 and observes other cores' relaxed
atomics. HIP code relies on that; Kokkos::volatile_load walks the
UnorderedMap insert list with it after the writer's __threadfence() plus
relaxed CAS. SPIR-V's OpLoad with the Volatile memory operand only means
the access cannot be eliminated, duplicated or combined, and IGC serves
it from L1, so on Aurora PVC hip.UnorderedMap_insert reports 901 keys
for 900.
HipLowerVolatileAccessesPass rewrites every volatile load and store of a
32 or 64 bit integer, float or pointer through a global or generic
pointer into `load atomic volatile ... syncscope("device") monotonic`
(floats and pointers through the same-width integer), which both SPIR-V
producers emit as OpAtomicLoad / OpAtomicStore with Relaxed semantics at
Device scope. The explicit "device" syncscope matters: the translator at
LLVM 17 hardcodes Device for atomic loads, newer ones map the default
scope to CrossDevice. 8 and 16 bit values, private, constant and
work-group local objects, under-aligned accesses and accesses that are
already atomic are left as they are.
…ixture volatile-accesses.ll holds one kernel of accesses HipLowerVolatileAccessesPass must rewrite (32 and 64 bit integers, floats and pointers through global and generic pointers) and one of accesses it must leave alone (8 and 16 bit, vectors, work-group local, private, constant, under-aligned, already atomic). The runner drives the standalone hip-lower-volatile-accesses pass with opt, counts both sets, expects the under-aligned warnings, and translates the result to SPIR-V through spirv-val.
…ss SPIR-V The Khronos translator built from LLVM 21 keeps an OpName on the kernel body and emits the OpEntryPoint as a separate wrapper function whose only instruction is an OpFunctionCall to it, so the entry point id in the disassembly is neither numeric nor the function that holds the accesses. TestFixVolatileLoadLoweringSPIRV looked up the body by a numeric entry point id and counted 0 OpAtomicLoad / 0 OpAtomicStore on the pocl 7.1 (llvm-21) CI lane. Accept any id and follow a wrapper's OpFunctionCall to the callee before counting.
Collaborator
Author
|
/run-aurora-ci |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CUDA gives a volatile access to global memory the meaning of PTX
ld.volatile/st.volatile, a relaxed system scope access that bypasses L1, and HIP code such asKokkos::volatile_loadrelies on it. chipStar emitted such accesses as plainOpLoad/OpStorewith theVolatilememory operand, which only forbids eliminating or merging the access, so IGC serves it from L1 and readers see stale data.The new
HipLowerVolatileAccessesPassrewrites every volatile 32 or 64 bit load and store through a global or generic pointer into a relaxedsyncscope("device")atomic, which both SPIR-V producers emit asOpAtomicLoad/OpAtomicStore. Sub word, vector, private, constant, work group local, under aligned and already atomic accesses are left alone. Comes with a runtime reproducer, a SPIR-V inspection test and anoptfixture covering every rule of the pass.Found with the Kokkos
hip.UnorderedMap_inserttest on Aurora PVC, which reported 901 keys inserted for 900.Fixes #1508