feat(sd3): dump-verified boundary input dtypes + fp32 OCR recipe - #93
Draft
Rockdu wants to merge 47 commits into
Draft
feat(sd3): dump-verified boundary input dtypes + fp32 OCR recipe#93Rockdu wants to merge 47 commits into
Rockdu wants to merge 47 commits into
Conversation
…instead of ignored_params
…n_args and slim compiler output types
…on module docstring
…el and boundary validation
… policy and ZeRO-2 changes
…nd most-specific annotation wins
…est-first wrap plan
…he rest of rule validation
…ng sglang's per-pipeline default
…ap_plan off the result
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
configs/sd3.py: SD3 opts into the boundary input dtype policy —{"latents": "default", "cond": None, "timestep": "fp32"}(base default is full passthrough after [Do Not Merge] refactor(fsdp)(4/5): per-parameter weight-precision control for FSDP2 #91).scripts/run-diffusion-grpo-sd3-ocr-sglang.sh:--precision-default-dtype fp16→fp32.arguments.py+sglang_diffusion_engine.py: forward--sglang-text-encoder-precisionsto the rollout engine (same explicit-forward pattern asdit_precision); the recipe passesfp32 fp32 fp32. With an fp32 DiT there is no autocast on either side, so the fp16 CLIPpooled_projectionshard-crashes the engine (mat1 and mat2 must have the same dtype) — the text encoders must run fp32 too.precision.py: fix a [Do Not Merge] refactor(fsdp)(4/5): per-parameter weight-precision control for FSDP2 #91 regression —wrap_planlooks upgather_dtypes(resolved on clean pre-LoRA FQNs by design) with post-LoRA FQNs, so every--use-lorarecipe dies withKeyError: 'base_model.model.transformer_blocks.0'at FSDP wrap. Strip the PEFT prefix at the lookup. Candidate for cherry-pick into [Do Not Merge] refactor(fsdp)(4/5): per-parameter weight-precision control for FSDP2 #91; I will drop it here once it lands there.Why
--fsdp-master-dtypealready defaults to fp32; this flips the unified knob (train forward/gather + rollout dit) to fp32. With a fp32 train forward the LoRA IPC sync keepsSGLANG_DIFFUSION_LORA_MERGE_FP32=1, closing the last precision gap in the weight-sync path, and the deterministic e2e standard is anchored on fully fp32 numerics.Validation
record-e2e-standards.ymlon theh200/3gpurunner class (first run 30895256691 hit the wrap-plan KeyError; re-dispatched as 30896211680, which then hit the fp16-CLIP-vs-fp32-DiT engine crash; third dispatch 30897866080 with fp32 text encoders succeeded); the recorded standard is committed. Step-1train/log_prob_mean_abs_diff= 4.8e-08 (~20x tighter than the fp16-era standard), the parity read before any optimizer step.Files
miles/backends/fsdp_utils/configs/sd3.py— SD3input_dtype_policyopt-in (one comment line + one literal).scripts/run-diffusion-grpo-sd3-ocr-sglang.sh— recipe precision default fp16 → fp32.miles/backends/fsdp_utils/precision.py— wrap-plan gather-dtype lookup through the PEFT prefix (one line).miles/utils/arguments.py+miles/backends/sglang_diffusion_utils/sglang_diffusion_engine.py— text-encoder precision passthrough (2 lines).Checklist
pre-commit run --all-files— green in PR CItests/fast/backends/fsdp_utils/test_input_dtype_policy.py(registered, stage-a-cpu), a config literal needs no extra testpython3 train_diffusion.py --helpparses (verified on H200 devbox);--sglang-text-encoder-precisionsis an existing sglang CLI arg surfaced by the auto-prefix, now forwarded to the engine