Skip to content

chore(wandb): report GPU system metrics (SM Active) on secondaries - #2

Closed
Rockdu wants to merge 1 commit into
mainfrom
chore/wandb-system-metrics
Closed

chore(wandb): report GPU system metrics (SM Active) on secondaries#2
Rockdu wants to merge 1 commit into
mainfrom
chore/wandb-system-metrics

Conversation

@Rockdu

@Rockdu Rockdu commented Jul 22, 2026

Copy link
Copy Markdown
Owner

What

  • Bump wandb from 0.23.10.28.1.
  • Stop disabling system-stats collection on wandb secondaries (drop
    x_disable_stats=True in init_wandb_secondary).

Together these make the W&B System panel report the full per-GPU metric
set — including the NVML GPM family (GPU SM Active %, tensor/FP-pipe
activity, DRAM bandwidth, PCIe/NVLink throughput) — bringing us to parity with
miles core.

Why

Our System panel showed far fewer metrics than miles core (e.g. no
GPU SM Active (%)). Two independent causes, both fixed here:

  1. Version. SM Active and the rest of the NVML GPM profiling metrics for
    Hopper+ GPUs are collected by wandb-core without a DCGM daemon, but the
    feature only landed in wandb v0.26.0 (2026-04-13, feat: add NVML GPM profiling metrics for Hopper+ GPUs to system monitor wandb/wandb#11622).
    We were pinned to 0.23.1 (2025-12-03), which predates it by ~4 months, so
    the collection path simply did not exist. miles core leaves wandb
    unpinned and floats to a version that has it.

  2. Config. GPM metrics are sampled on the process that owns the GPU. In
    our Ray setup the wandb primary is the CPU-only driver
    (train_diffusion.py), which has no GPU; every GPU worker inits as a
    secondary, and those secondaries were forcing x_disable_stats=True
    so no process was ever sampling GPU stats. miles core never sets this flag.

Only two single processes init as secondary — the training actor at global
rank 0 and the rollout manager — so re-enabling stats does not create
duplicate per-node sampling.

Note: SM Active requires Hopper+ (H100/H200 and newer) hardware, since
NVML GPM is only available there.

Files

  • requirements.txt — bump wandb==0.23.1wandb==0.28.1.
  • miles/utils/wandb_utils.py — drop x_disable_stats=True from the secondary
    settings; add a comment explaining why stats stay on.

Checklist

  • pre-commit run --files ... passes on the changed files (yaml/ruff/autoflake/isort/format all green)
  • Added/updated tests for new behaviour — no tests added; this repo has no wandb test module and the change is init-time W&B Settings wiring that can't be meaningfully unit-tested without a live wandb backend + GPU
  • pytest -x is green — not run (no relevant tests; change is not unit-testable offline)
  • If launch flags changed, python3 train.py --help still parses — n/a, no launch flags changed
  • If a public flag was added, it appears in the CLI reference docs — n/a, no public flag added
  • If an example was added, it has a real walkthrough — n/a, no example added
  • Full verification requires an online run on H100/H200 to confirm GPU SM Active (%) and the rest of the GPM family appear in the System panel — not run (needs GPU + W&B backend)

The System panel logged far fewer metrics than miles core — notably no
GPU SM Active (%). Two causes, both fixed:

- wandb 0.23.1 predates the NVML GPM profiling metrics (SM/tensor/FP-pipe
  activity, DRAM/NVLink throughput) for Hopper+ GPUs, added in wandb 0.26.0
  (wandb/wandb#11622); bump to 0.28.1.
- GPM stats are sampled on the GPU-owning process, but every GPU worker inits
  as a wandb secondary with x_disable_stats=True while the primary is the
  CPU-only Ray driver, so nothing ever sampled GPU stats; drop the flag to
  match miles core. Only the rank-0 actor and rollout manager init as
  secondary, so this adds no duplicate per-node sampling.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Rockdu

Rockdu commented Jul 22, 2026

Copy link
Copy Markdown
Owner Author

Superseded by a draft PR against the upstream radixark/miles_diffusion repo.

@Rockdu Rockdu closed this Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant