[Question]: In exclusive DCI CTA mapping, what prevents two concurrently resident kernels with the same blockIdx from touching the same DCI and racing on atomicAdd_block?

[goelayu]

Question

In exclusive DCI CTA mapping, what prevents two concurrently resident kernels with the same blockIdx from touching the same DCI and racing on atomicAdd_block?

cc @benhg

[benhg]

Hi @goelayu , it is not safe to use exclusive DCI CTA mapping with multiple co-scheduled kernels. They will in fact race with AtomicAdd_block. This is why we provide the NVSHMEM_IBGDA_DCI_MAP_BY=sm option that uses physical SM ID which will never conflict.

The reason we have the less save MAP_BY CTA default today is for performance reasons. I believe we use GPU scoped atomics for RCs.

Will the MAP_BY SM not work with your use case ? Is it giving unacceptable performance?