[AIROCMLIR-1061] Increase rocm-smi timeout#2428
Conversation
There was a problem hiding this comment.
Pull request overview
This PR reduces flakiness in tuningRunner-* LIT tests by increasing the rocm-smi GPU topology discovery timeout, aligning rocMLIR with the previously proven change in rocmlirTriton.
Changes:
- Increase
rocm-smiinvocation timeout from 10s to 60s during GPU topology discovery. - Add an inline comment explaining why a larger timeout is necessary on large multi-GPU systems.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Verdict: APPROVE -- submitted as COMMENT (automated reviews are advisory) · Findings: 0 (0 Critical, 0 Major, 0 Minor)
Scope
Bumps the rocm-smi subprocess timeout in GpuTopology.discover() from 10s to 60s and adds an explanatory comment, addressing flaky tuningRunner-* LIT tests. Single-file, single-line functional change (mlir/utils/performance/tuningRunner.py), porting a fix already validated in rocmlirTriton.
Findings
No blocking issues found.
Notes
- The added comment cites ~20s enumeration time while the timeout is set to 60s; the extra headroom is reasonable given the flakiness motivation, so no change requested.
- Change is a pure constant bump on an existing, tested code path; no new op/pass/API surface, so no additional Lit/E2E coverage is warranted.
CI status
No non-self checks report fail or cancel. Python format/lint and Python performance script tests pass; C/C++ premerge is still in progress at review time.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #2428 +/- ##
===========================================
+ Coverage 82.57% 82.86% +0.30%
===========================================
Files 120 120
Lines 42852 42828 -24
Branches 7110 7106 -4
===========================================
+ Hits 35381 35489 +108
+ Misses 4815 4696 -119
+ Partials 2656 2643 -13
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Motivation
This PR addresses some flaky/intermittent behavior observed in the
tuningRunner-*LIT tests when calling intorocm-smifor GPU topology discovery.Technical Details
In https://github.com/ROCm/rocmlirTriton/pull/310 in rocmlirTriton we bumped the timeout from 10s -> 60s. Looking at the history of nightly CI runs since then shows that this issue has not come up. This PR just ports those same changes to rocMLIR.
Test Plan
Test Result
Submission Checklist