feat(ltx): consume rollout sigmas exactly and train with fp32 master weights - #94
Draft
Rockdu wants to merge 50 commits into
Draft
feat(ltx): consume rollout sigmas exactly and train with fp32 master weights#94Rockdu wants to merge 50 commits into
Rockdu wants to merge 50 commits into
Conversation
…instead of ignored_params
…n_args and slim compiler output types
…on module docstring
…el and boundary validation
… policy and ZeRO-2 changes
…nd most-specific annotation wins
…est-first wrap plan
…he rest of rule validation
…ng sglang's per-pipeline default
…ap_plan off the result
Rockdu
force-pushed
the
feat/ltx-precision-policy
branch
from
August 4, 2026 18:37
5d55405 to
beb9ce4
Compare
Rockdu
force-pushed
the
feat/ltx-precision-policy
branch
from
August 4, 2026 20:45
8010543 to
b940e5a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
PreparedBatchgains asigmasfield (exact-matched from the rollout snapshot) andcompute_noise_predreceives bothtimesteps_inputandsigmas_input; each family reads the domain its model consumes (sd3/wan → t, ltx/qwen → σ). Theneeds_timestep_scalingknob and its rescaling branches are deleted.--fsdp-master-dtype fp32and re-record its e2e standard.base_model.model.prefix (KeyErroron every LoRA run otherwise).Why
t_hostand the σ the rollout DiT consumed (t_device) are computed separately in sglang-d and sit 1–2 fp32 ULPs apart, so no rescaling of one recovers the other — each domain's exact original is a different array. Dividing t back down was the last known timestep gap (LTX AdaLN input rel 1.8e-3 → now bit-exact).dit_trajectory(sglang #32683, miles refactor(diffusion): require rollout scheduler sigmas, drop timesteps-derived fallbacks #92) — present on both mains.Validation
Paired-dump alignment (LTX-2.3, 4×H200, latent bit-exact pairing, 2580 tensor pairs):
With attention kernels pinned to the same flash path on both sides (
--fsdp-attention-backend sdpa_flash+ the rollout patches here) the forward reaches 2580/2580 bit-exact pairs — full train↔rollout forward parity. The production recipe keepssdpa_math(deterministic backward), whose residual is pure MATH-vs-flash kernel noise. Two rollout parity patches (ltxpatch group): drop all-true attention masks (a kept trivial mask disqualifies flash and lands on cuDNN — torch's Hopper default priority is cuDNN-first) and run the output LayerNorm tail in fp32 with a single rounding at proj_out, matching autocast's rounding points.Train metrics reproduce bitwise across two machines and sglang builds (
model_output_mean_abs_diff=4.362646e-03,log_prob_mean_abs_diff=4.172325e-07). The re-recorded standard shows the same shift in CI: step-0 rollout reward identical,log_prob_mean_abs_diff1.0e-6 → 3.6e-7; grad_norm moves with the fp32 optimizer path.Files
miles/backends/fsdp_utils/loss_hub/types.py—PreparedBatch.timesteps/sigmas, one domain eachmiles/backends/fsdp_utils/loss_hub/flow_grpo.py—_sigmas_for_timestepsexact-match lookup; branch removedmiles/backends/fsdp_utils/loss_hub/nft.py— same rename; field-domain mixup fixed (timestepscarried σ)miles/backends/fsdp_utils/configs/train_pipeline_config.py—compute_noise_pred(..., sigmas_input); knob deletedmiles/backends/fsdp_utils/configs/{ltx,qwen_image,sd3,wan2_2}.py— families read their model's domainmiles/backends/fsdp_utils/actor.py— passes both through the input dtype policymiles/backends/fsdp_utils/precision.py— wrap-plan LoRA-prefix fix (refactor(fsdp)(4/5): per-parameter weight-precision control for FSDP2 #91 bug)scripts/run-diffusion-grpo-ltx23-sglang.sh— fp32 master weightstests/ci/fixtures/e2e_standards/test_ltx23_pickscore_grpo_4xGPU.json— re-recorded on the h200-5gpu runnertests/fast/backends/fsdp_utils/test_flow_grpo_sigma_lookup.py,test_precision_plan.py,test_input_dtype_policy.py,configs/test_train_pipeline_config_registry.py— new/updated CPU tests (stage-a-cpu)Checklist
pre-commit runpasses on the touched files (black/ruff/isort/autoflake, dev image)pytest tests/fast/backends/fsdp_utils tests/fast/utils -qgreen (121 passed; precision tests re-run after the final rebase)python3 train.py --helpstill parses — no flag changes (recipe value only)