Skip to content

feat(sd3): dump-verified boundary input dtypes + fp32 OCR recipe - #93

Draft
Rockdu wants to merge 47 commits into
mainfrom
feat/sd3-boundary-input-policy
Draft

feat(sd3): dump-verified boundary input dtypes + fp32 OCR recipe#93
Rockdu wants to merge 47 commits into
mainfrom
feat/sd3-boundary-input-policy

Conversation

@Rockdu

@Rockdu Rockdu commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #91 (feat/fsdp-precision-control) — only the last 2 commits are new; will rebase onto main once #91 merges.

What

Why

  • Boundary opt-in: the trajectory serializes latents as fp32 while the SD3 rollout DiT consumed dit-precision; under the passthrough default the train entry dtype diverges from the rollout forward. The policy mirrors what the rollout actually feeds, dump-verified on paired GRPO runs: 2680/2680 tensors bit-exact (cond + uncond CFG branches), 0 dtype mismatches, comparator 2680 passed / 0 failed. Casting the timestep instead rounds the value itself (fp16(857.692) == 857.5); the merged CLIP+T5 embeds enter uncast (T5 runs fp32).
  • fp32 recipe: --fsdp-master-dtype already defaults to fp32; this flips the unified knob (train forward/gather + rollout dit) to fp32. With a fp32 train forward the LoRA IPC sync keeps SGLANG_DIFFUSION_LORA_MERGE_FP32=1, closing the last precision gap in the weight-sync path, and the deterministic e2e standard is anchored on fully fp32 numerics.

Validation

  • The wrap-plan KeyError reproduced and the fix verified with a minimal CPU repro (bare and PEFT-wrapped module trees) on torch 2.x.
  • Paired-dump alignment runs (fp16, LoRA r=32 gaussian, 10-step CFG, SDE candidate step 3, 2×H200): train↔rollout bit-exact end to end with the opt-in policy; entry dtypes hidden_states fp16/fp16, timestep fp32/fp32, encoder_hidden_states fp32/fp32.
  • SD3 OCR e2e standard re-record for the fp32 recipe: dispatched via record-e2e-standards.yml on the h200/3gpu runner class (first run 30895256691 hit the wrap-plan KeyError; re-dispatched as 30896211680, which then hit the fp16-CLIP-vs-fp32-DiT engine crash; third dispatch 30897866080 with fp32 text encoders succeeded); the recorded standard is committed. Step-1 train/log_prob_mean_abs_diff = 4.8e-08 (~20x tighter than the fp16-era standard), the parity read before any optimizer step.

Files

  • miles/backends/fsdp_utils/configs/sd3.py — SD3 input_dtype_policy opt-in (one comment line + one literal).
  • scripts/run-diffusion-grpo-sd3-ocr-sglang.sh — recipe precision default fp16 → fp32.
  • miles/backends/fsdp_utils/precision.py — wrap-plan gather-dtype lookup through the PEFT prefix (one line).
  • miles/utils/arguments.py + miles/backends/sglang_diffusion_utils/sglang_diffusion_engine.py — text-encoder precision passthrough (2 lines).

Checklist

  • pre-commit run --all-files — green in PR CI
  • Added/updated tests — none added; the policy mechanism is covered by tests/fast/backends/fsdp_utils/test_input_dtype_policy.py (registered, stage-a-cpu), a config literal needs no extra test
  • CPU suites and stage-b/c GPU e2e (incl. SD3 vs the new fp32 standard, bit-for-bit) — all green in PR CI
  • python3 train_diffusion.py --help parses (verified on H200 devbox); --sglang-text-encoder-precisions is an existing sglang CLI arg surfaced by the auto-prefix, now forwarded to the engine
  • CLI reference docs — n/a, no new flag
  • Example walkthrough — n/a, no new example

Rockdu added 30 commits August 3, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant