chore(wandb): report GPU system metrics (SM Active) on secondaries - #2
Closed
Rockdu wants to merge 1 commit into
Closed
chore(wandb): report GPU system metrics (SM Active) on secondaries#2Rockdu wants to merge 1 commit into
Rockdu wants to merge 1 commit into
Conversation
The System panel logged far fewer metrics than miles core — notably no GPU SM Active (%). Two causes, both fixed: - wandb 0.23.1 predates the NVML GPM profiling metrics (SM/tensor/FP-pipe activity, DRAM/NVLink throughput) for Hopper+ GPUs, added in wandb 0.26.0 (wandb/wandb#11622); bump to 0.28.1. - GPM stats are sampled on the GPU-owning process, but every GPU worker inits as a wandb secondary with x_disable_stats=True while the primary is the CPU-only Ray driver, so nothing ever sampled GPU stats; drop the flag to match miles core. Only the rank-0 actor and rollout manager init as secondary, so this adds no duplicate per-node sampling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
|
Superseded by a draft PR against the upstream radixark/miles_diffusion repo. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
wandbfrom0.23.1→0.28.1.x_disable_stats=Trueininit_wandb_secondary).Together these make the W&B System panel report the full per-GPU metric
set — including the NVML GPM family (GPU SM Active %, tensor/FP-pipe
activity, DRAM bandwidth, PCIe/NVLink throughput) — bringing us to parity with
milescore.Why
Our System panel showed far fewer metrics than
milescore (e.g. noGPU SM Active (%)). Two independent causes, both fixed here:Version.
SM Activeand the rest of the NVML GPM profiling metrics forHopper+ GPUs are collected by wandb-core without a DCGM daemon, but the
feature only landed in wandb v0.26.0 (2026-04-13, feat: add NVML GPM profiling metrics for Hopper+ GPUs to system monitor wandb/wandb#11622).
We were pinned to
0.23.1(2025-12-03), which predates it by ~4 months, sothe collection path simply did not exist.
milescore leaveswandbunpinned and floats to a version that has it.
Config. GPM metrics are sampled on the process that owns the GPU. In
our Ray setup the wandb primary is the CPU-only driver
(
train_diffusion.py), which has no GPU; every GPU worker inits as asecondary, and those secondaries were forcing
x_disable_stats=True—so no process was ever sampling GPU stats.
milescore never sets this flag.Only two single processes init as secondary — the training actor at global
rank 0 and the rollout manager — so re-enabling stats does not create
duplicate per-node sampling.
Note:
SM Activerequires Hopper+ (H100/H200 and newer) hardware, sinceNVML GPM is only available there.
Files
requirements.txt— bumpwandb==0.23.1→wandb==0.28.1.miles/utils/wandb_utils.py— dropx_disable_stats=Truefrom the secondarysettings; add a comment explaining why stats stay on.
Checklist
pre-commit run --files ...passes on the changed files (yaml/ruff/autoflake/isort/format all green)Settingswiring that can't be meaningfully unit-tested without a live wandb backend + GPUpytest -xis green — not run (no relevant tests; change is not unit-testable offline)python3 train.py --helpstill parses — n/a, no launch flags changedGPU SM Active (%)and the rest of the GPM family appear in the System panel — not run (needs GPU + W&B backend)