fix(cuda.core): preserve ordering for asynchronous memory operations in tests - #2657
Conversation
Ensure the exporting process completes asynchronous allocation and initialization before an importing child accesses the shared buffer.
This comment has been minimized.
This comment has been minimized.
Require PatternGen callers and affected examples/tests to preserve stream ordering, adding explicit synchronization only across host and IPC boundaries.
There was a problem hiding this comment.
LGTM — but please see the one inline suggestion (it was classified as "Low").
codex gpt-5.6-sol ultra:
The original issue is now fixed properly:
- Allocation and
PatternGeninitialization use the same stream. - That stream is synchronized before the child process starts.
- All 33
PatternGencall sites now provide an explicit stream, making recurrence less likely. - Host accesses and CuPy interoperability are likewise ordered correctly.
- A repo-wide sweep found no analogous missed allocation/use race.
The rationale is technically sound. CUDA documents that cross-stream access to a stream-ordered allocation must be explicitly ordered, that nonblocking streams do not implicitly synchronize with the legacy default stream, and that importing an IPC allocation does not wait for the exporter’s allocation to become ready. A host-side stream synchronization before process handoff is a simple, valid test-level barrier. CUDA allocator documentation (https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/stream-ordered-memory-allocation.html), default-stream behavior (https://docs.nvidia.com/cuda/cuda-driver-api/stream-sync-behavior.html).
|
Summary
Prevent races caused by allocating memory asynchronously on one CUDA stream and consuming it on another stream or outside CUDA stream ordering. This expands the original IPC fix to cover affected memory tests and examples.
Several tests incorrectly assumed that memory allocations against a default stream behaved synchronously. The legacy default stream has special synchronization semantics with respect to blocking streams, but nothing orders it against non-blocking streams. This PR fixes several latent races related to that.
Changes
PatternGencallers to provide a stream and submit fills, copies, and verification on that stream.Related Work
Follow-up to #1308.
Checklist