[xpupti] Add XCCL collective communication activity tracing to XPU plugin#1396
[xpupti] Add XCCL collective communication activity tracing to XPU plugin#1396tsocha wants to merge 11 commits into
Conversation
Enable PTI_VIEW_COMMUNICATION collection in the XPU PTI plugin so oneCCL host-side collective operations show up in Kineto traces. Events are emitted as COLLECTIVE_COMM activities named with an "xccl::" prefix and carry the PTI communicator id as metadata. - Gate new code paths on PTI_VERSION_AT_LEAST(0, 17) - Wire enable/disable of PTI_VIEW_COMMUNICATION in XpuptiActivityApi - Add handleCommunicationActivity for pti_view_record_comms records - Add unit tests covering naming, field mapping, and out-of-range drop - Document INTEL_LIBITTNOTIFY64 requirement in libkineto/README.md
|
@gujinghui please review it. |
| The default trace output is a JSON file that can be visualized in Chrome Trace Viewer or Perfetto. The trace output is generated by the `ChromeTraceLogger` instance. The `ChromeTraceLogger` writes to a JSON file using `std::ofstream` in `output_json.cpp` to maximize performance during export. This instance is created by the `ActivityProfilerController` and is stored in the `ActivityLoggerFactory` alongside its protocol. Using this schema, Kineto supports multiple trace output formats. | ||
|
|
||
| - Intel XCCL: to enable collecting of oneCCL host events, `INTEL_LIBITTNOTIFY64` enviroment variable have to be set as path to `pti_view.so` location. | ||
|
|
There was a problem hiding this comment.
Why does this need to be in the general instructions? Is this something that's covered in Intel's other docs? I'd like to keep this file short and general.
There was a problem hiding this comment.
This env variable is required by ITT which is used by PTI to collect these events.
Without this variable user won't see oneCCL events in his trace.
I wanted to expose this information to avoid confusion of the Kineto user.
We are working to remove this requirement in the future but due to performance requirements of PTI integration it's not ready yet.
I could create a new README file inside xpupti plugin directory but I'm afraid that it will be hidden.
What do you think?
There was a problem hiding this comment.
@tsocha Do we have public doc to explain this env var?
There was a problem hiding this comment.
There is some documentation related to V-tune and ITT:
https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2025-4/attaching-itt-apis-to-a-launched-application.html
Unfortunately it's not directly related to Kineto.
In our case the PTI is a collector.
There is also a documentation of tests for ITT:
https://github.com/intel/ittapi/blob/master/src/ittnotify_refcol/README.md
Again, it's at least 2 levels lower than Kineto and we should not expect that user will know that he need to set an env variable for ITT -> which is used by PTI -> which is used by Kineto.
The best documentation I was able to get is PTI Known Issues section of the documentation:
https://intel.github.io/pti-gpu/knownissues.html
I want to avoid confusion from the user.
If the main README.md is not a good place for it, maybe we should create another one in XPUPTI plugin directory.
There was a problem hiding this comment.
@scotts
README.md is clean again. Now, we emit a runtime warning.
There was a problem hiding this comment.
@scotts Can you please review this PR. as it said README.md is clean
and all additional requirements will be documented at Intel PTI special page in a way specific to PyTorch profiler
thanks!
att @gujinghui
|
@scotts please help review this PR. Thanks. |
tsocha
changed the title
|
@scotts @divyanshk @ryanzhang22 |
5 participants
It's a part of #1335 3/3
Enable PTI_VIEW_COMMUNICATION collection in the XPU PTI plugin so oneCCL host-side collective operations show up in Kineto traces. Events are emitted as COLLECTIVE_COMM activities named with an "xccl::" prefix and carry the PTI communicator id as metadata.