Skip to content

test(e2e): re-record the SD3.5 OCR standard after sglang 585a7d05e - #88

Merged
Rockdu merged 1 commit into
mainfrom
fix/rerecord-sd3-ocr-sglang-drift
Aug 3, 2026
Merged

test(e2e): re-record the SD3.5 OCR standard after sglang 585a7d05e#88
Rockdu merged 1 commit into
mainfrom
fix/rerecord-sd3-ocr-sglang-drift

Conversation

@Rockdu

@Rockdu Rockdu commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

What

Re-record tests/ci/fixtures/e2e_standards/test_sd3_ocr_grpo_2xGPU.json. No code changes. Pairs with #89, which removes the miles-side sigmas fallback (to recalc by timesteps) so this class of silent drift cannot recur.

Why

The committed standard (recorded 2026-07-27) encodes values from a derived sigma schedule. CI installs sglang from unpinned main; sglang 585a7d05e (#32683, bisected in 6 rounds with byte-diffed recording artifacts) started returning the scheduler's own sigmas snapshot in dit_trajectory, flipping miles' if "scheduler_sigmas" in ... branch away from the cat([timesteps / 1000, zeros(1)]) fallback. stage-c-3-gpu-h200 has been red on main itself since then.

The snapshot is strictly more accurate than the fallback, so the new values are the correct ones to record (probe of real tensors, run 30778705251):

timesteps: n=11  last3=[278.0487976074219, 8.928571701049805, 0.0]
sigmas   : n=11  last3=[0.2780487835407257, 0.008928571827709675, 0.0]
fallback : n=12  last3=[0.008928571827709675, 0.0, 0.0]
  • fp32 sigma × 1000 → ÷ 1000 round-trip loses low-order bits (rel. 1.42e-8) — 1000 is not a power of two, so both roundings are lossy.
  • traj.timesteps is [T+1] (already ends in 0.0), so the fallback's extra cat duplicates the terminal zero: 12 entries vs the snapshot's 11.
  • First divergence is train/log_prob_new_idx_0[0]: -0.7209256328642368 → -0.720925610512495 (2.2e-8); grad_norm and rollout-1 rewards cascade from there.

Controls

  • Pinning pre-drift sglang 3312645a3 reproduces the committed standard 8/8 (run 30770528080).
  • Nulling dit_trajectory.sigmas on current sglang (forcing the old fallback) also reproduces it 8/8 (run 30772902308) — the sigmas snapshot accounts for all of the drift.
  • The re-recorded values are byte-identical when recorded on main (run 30766360335) and on an unrelated branch (run 30766005058), and match refactor(diffusion): require rollout scheduler sigmas, drop timesteps-derived fallbacks #89's stage-c observations bitwise.

Known side effect of the same upstream change (tracked separately, not in this PR): DiffusionNFT's kept training timesteps went 10 → 9, because resolve_nft_sigmas trims by int(numel * 0.99) and the array length changed.

Files

File Role
tests/ci/fixtures/e2e_standards/test_sd3_ocr_grpo_2xGPU.json Re-recorded standard. Only file touched.

Checklist

  • pre-commit run --all-files passes — run on the touched file
  • Added/updated tests for new behaviour — n/a, re-records an existing test's standard
  • pytest -x is green — n/a, no code change
  • If launch flags changed, python3 train.py --help still parses — n/a
  • If a public flag was added, it appears in the CLI reference docs — n/a
  • If an example was added, it has a real walkthrough — n/a

@Rockdu
Rockdu marked this pull request as draft August 2, 2026 22:35
@Rockdu Rockdu changed the title test(e2e): re-record the SD3.5 OCR standard after sglang drift [DRAFT] test(e2e): re-record the SD3.5 OCR standard (cause not yet identified) Aug 2, 2026
@Rockdu Rockdu changed the title [DRAFT] test(e2e): re-record the SD3.5 OCR standard (cause not yet identified) test(e2e): re-record the SD3.5 OCR standard after sglang 585a7d05e Aug 3, 2026
@Rockdu
Rockdu marked this pull request as ready for review August 3, 2026 07:24
@Rockdu
Rockdu merged commit 466fc1b into main Aug 3, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant