[CI] Parity auto-trigger: open up mi300/mi200/navi31 collection#3397
Open
ethanwee1 wants to merge 2 commits into
Open
[CI] Parity auto-trigger: open up mi300/mi200/navi31 collection#3397ethanwee1 wants to merge 2 commits into
ethanwee1 wants to merge 2 commits into
Conversation
Auto-parity was trunk-scoped (mi350 only). mi300/mi200/navi31 run in their own scheduled upstream workflows on their own SHAs at a different cadence, so the trunk-push scan never reached them. - Discover candidate SHAs from both trunk.yml pushes and the scheduled per-arch workflows (fetch_scheduled_commits / fetch_candidate_commits), deduped newest-first; a 40-hex guard drops stray non-row output. - Hold back SHAs newer than the newest scheduled run so a lagging mi300/mi200/navi batch can still join that SHA's report, yielding ONE combined parity report per SHA (parity.yml's matrix already emits a per-arch artifact plus a merged summary) once every arch that ran has finished. When no scheduled runs exist, nothing is held (mi350/trunk behaves exactly as before). - Expand ARCHS_IN to mi350 mi300 mi200 navi31. - Per-config CUDA-baseline gating: a SHA whose CUDA jobs for a test config did not run (e.g. a failed trunk run that never launched CUDA default) has no baseline for that config, so exclude just that config from the dispatch (via parity.yml's exclude_* inputs) instead of emitting a bogus all-MISSED column, and drop its ROCm shards from the completion gate. No parity_job_config.json change: mi300 has no fallback; its default/ distributed/inductor come from rocm-mi300 / periodic-rocm-mi300 / inductor-rocm-mi300, which run together on one scheduled SHA.
|
Jenkins build for 639f16e2d55b12f21a3892f7887fbfcda85c0306 commit finished as FAILURE |
Shard counts were hardcoded (config value or literals), so when an arch falls back to a different workflow whose sharding differs, the constructed "(config, i, N)" job/artifact keys miss the real jobs. Concretely mi200's default/inductor workflows are dormant upstream and fall back to trunk-rocm-sandbox, which shards default into 10 (config says 6) and inductor into 4 (config says 2) - so every key "DOES NOT EXIST IN JOBS" and the mi200 shard of a combined auto report fails. Add derive_shard_count(), which reads the actual shard total from the resolved run's job names, and use it for ROCm default/distributed/inductor (falling back to the config value only when no matching jobs are found). This unblocks mi200 in the multi-arch auto-trigger.
|
Jenkins build for 6b0eed6a8fa73fb298d5c6372b61ba858fb374dc commit finished as ABORTED |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Auto-parity was trunk-scoped — mi350 is the only ROCm arch that rides along in
trunk.yml, so the candidate-SHA scan (completedtrunk.ymlpushes) never reached mi300/mi200/navi31, which run in their own scheduled upstream workflows on their own SHAs at a different cadence. This opens up those arches so we collect their full runs (not just trunk), producing one combined parity report per SHA with a per-arch artifact plus a merged summary.What changed (
.github/workflows/parity-auto.ymlonly)trunk.ymlpushes, mine SHAs from the scheduled per-arch workflows (rocm-mi300/periodic-rocm-mi300/inductor-rocm-mi300, the mi200/navi equivalents, etc.). Merged newest-first, one row per SHA, with a 40-hex guard to drop stray output (a missing scheduled workflow returns a 404 body).ARCHS_IN=mi350 mi300 mi200 navi31.default) has no baseline for that config, so that config is excluded from the dispatch (viaparity.yml'sexclude_*inputs) rather than emitting a bogus all-MISSEDcolumn; its ROCm shards are also dropped from the completion gate. Existing per-check-run completion gating is unchanged.No
parity_job_config.jsonchange: mi300 has no fallback — its default/distributed/inductor come fromrocm-mi300/periodic-rocm-mi300/inductor-rocm-mi300, which run together on one scheduled SHA.Validation (on fork
ethanwee1/pytorch, this branch)e3fc47c5, whose mi300 distributed shards never completed).ba9910c6(full CUDA default/distributed/inductor baseline + full mi300 shards), mi300 sourced from its full workflows: overall 4.92% disagreement (28837842422).e3fc47c5, so its DEFAULT column is skewed — precisely the case the new gate now prevents from auto-dispatching.)Test plan
e3fc47c5).Update: mi200 shard-count fix (added)
Opening up mi200 surfaced a latent bug: shard counts were hardcoded, so when an arch falls back to a differently-sharded workflow the constructed
(config, i, N)keys miss the real jobs. mi200 default/inductor are dormant upstream and fall back totrunk-rocm-sandbox(10 default / 4 inductor shards vs config 6/2), which failed the mi200 shard of a combined auto run. Addedderive_shard_count()to read the real shard total from the resolved run; used for ROCm default/distributed/inductor.