Repair ROCm kernel stream attribution (#1416)#1416
Open
sanrise wants to merge 3 commits into
Open
Conversation
|
@sanrise has exported this pull request. If you are a Meta employee, you can view the originating Diff in D104473177. |
Contributor
Author
meta-codesync
Bot
changed the title
sanrise
added a commit
to sanrise/kineto
that referenced
this pull request
Jun 1, 2026
Summary: **Problem** ROCm Kineto traces can show kernels from independent HIP streams on the wrong stream, making unrelated worker activity look mixed together. **Why** ROCProfiler can deliver async kernel dispatch records where multiple kernels share one correlation and another runtime launch correlation is missing. Trusting correlation alone means Kineto can collapse or spread kernels onto the wrong stream. The repaired ROCm stream id is a 64-bit HIP stream value, and GPU user annotations also need to preserve that full value so Perfetto keeps annotations nested on the same lane as the kernels they cover. **Fix** Keep the HIP stream and demangled kernel name from runtime callbacks, carry dispatch thread and external stream data on async rows, and repair ambiguous duplicated kernel correlations only when runtime thread or canonical kernel name maps to exactly one stream. Preserve the full 64-bit resource id when creating generic GPU user annotations so `Stage::...` annotations stay aligned with repaired ROCm kernel lanes. The implementation limits overhead by building copy and kernel repair maps only for activity types requested by the trace, building the kernel-name map only when duplicated async kernel correlations are present, and tracking pushed stream correlations in cached kernel args instead of adding another thread-local hash set per kernel launch. If the evidence is ambiguous, Kineto preserves the original async stream instead of guessing. Differential Revision: D104473177
sanrise
force-pushed
the
export-D104473177
branch
from
a30fe3e to
e967a1e
Compare
Summary: The separate memcpy-kind report said the HIP runtime call knew the copy direction, but the GPU trace rendered the matching copy as D2D. Before changing Kineto, we needed a control experiment that removed Sigrid, Vanguard, batching, and model memory behavior from the picture. This sample runs plain HIP copies with explicit directions, prints the runtime source of truth, and then prints the Kineto runtime and GPU memcpy kinds side by side. Locally, the sample did not reproduce the D2D label: H2D stayed H2D and D2H stayed D2H. That negative result is useful because it tells us basic HIP copy-kind mapping works on this host, and if Vanguard still shows D2D then the bug likely needs a production-specific memory path such as staging, pointer classification, managed memory, or serving allocator behavior. Differential Revision: D104004242
Summary: The stream-interleaving report is about trust in the trace timeline: independent workers are expected to issue compute on separate streams, but the trace can make those workers look merged, shared, or unevenly represented. To isolate that symptom from the full serving stack, this sample creates a controlled workload with one host worker and one HIP stream per worker, then launches an identifiable kernel family from each worker. The program prints the HIP stream handles as the source of truth and summarizes the Kineto kernel stream IDs that come back. On this host, the 4-worker control maps cleanly to 4 Kineto streams, while the 8-worker stress case creates 8 HIP streams but Kineto reports only 4 kernel stream IDs with worker pairs sharing streams. That gives us a compact local reproduction of the Issue C class of symptom before chasing it inside Vanguard. Differential Revision: D104300974
Summary: **Problem** ROCm Kineto traces can show kernels from independent HIP streams on the wrong stream, making unrelated worker activity look mixed together. **Why** ROCProfiler can deliver async kernel dispatch records where multiple kernels share one correlation and another runtime launch correlation is missing. Trusting correlation alone means Kineto can collapse or spread kernels onto the wrong stream. The repaired ROCm stream id is a 64-bit HIP stream value, and GPU user annotations also need to preserve that full value so Perfetto keeps annotations nested on the same lane as the kernels they cover. **Fix** Keep the HIP stream and demangled kernel name from runtime callbacks, carry dispatch thread and external stream data on async rows, and repair ambiguous duplicated kernel correlations only when runtime thread or canonical kernel name maps to exactly one stream. Preserve the full 64-bit resource id when creating generic GPU user annotations so `Stage::...` annotations stay aligned with repaired ROCm kernel lanes. The implementation limits overhead by building copy and kernel repair maps only for activity types requested by the trace, building the kernel-name map only when duplicated async kernel correlations are present, and tracking pushed stream correlations in cached kernel args instead of adding another thread-local hash set per kernel launch. If the evidence is ambiguous, Kineto preserves the original async stream instead of guessing. Differential Revision: D104473177
sanrise
force-pushed
the
export-D104473177
branch
from
e967a1e to
0f8ef8c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Problem
ROCm Kineto traces can show kernels from independent HIP streams on the wrong stream, making unrelated worker activity look mixed together.
Why
ROCProfiler can deliver async kernel dispatch records where multiple kernels share one correlation and another runtime launch correlation is missing. Trusting correlation alone means Kineto can collapse or spread kernels onto the wrong stream. The repaired ROCm stream id is a 64-bit HIP stream value, and GPU user annotations also need to preserve that full value so Perfetto keeps annotations nested on the same lane as the kernels they cover.
Fix
Keep the HIP stream and demangled kernel name from runtime callbacks, carry dispatch thread and external stream data on async rows, and repair ambiguous duplicated kernel correlations only when runtime thread or canonical kernel name maps to exactly one stream. Preserve the full 64-bit resource id when creating generic GPU user annotations so
Stage::...annotations stay aligned with repaired ROCm kernel lanes. The implementation limits overhead by building copy and kernel repair maps only for activity types requested by the trace, building the kernel-name map only when duplicated async kernel correlations are present, and tracking pushed stream correlations in cached kernel args instead of adding another thread-local hash set per kernel launch. If the evidence is ambiguous, Kineto preserves the original async stream instead of guessing.Differential Revision: D104473177