Do not quantize the teacher by default in the distillation and GOLD entry points

[behroozazarkhalili]

What this does

Closes #6720.

The distillation and GOLD entry points default the teacher's quantization to the student's, so --load_in_4bit (or any other ModelConfig quantization flag) quantizes the teacher as well. This drops quantization_config from the teacher's default init kwargs, leaving the teacher in full precision unless the user asks otherwise.

The rationale is the one @cmpatino gave in #6642 (comment): the consensus in the literature is to avoid quantizing the teacher. The teacher defines the reference distribution the student is trained to match, so degrading it degrades the training signal itself, and since it receives no gradients the memory saving buys less than the same quantization does on the student.

Files, after the examples reorganization

This PR originally targeted examples/scripts/distillation.py, examples/scripts/gkd.py and examples/scripts/gold.py. #6820 reorganized examples into per-example folders and #6765 moved the distillation entry point, so those paths no longer exist. The branch has been merged with main and now targets the two surviving call sites:

• trl/scripts/distillation.py
• examples/gold_chatbot_arena/gold_chatbot_arena.py

gkd.py has no surviving equivalent that loads a teacher locally, so nothing replaces that hunk.

In trl/scripts/distillation.py the change also drops one sentence from an existing comment, which said the teacher carries quantization through teacher_model_init_kwargs. That described the coupled default this PR removes, so it is no longer accurate.

Why no new config field

teacher_model_init_kwargs already exists on both configs and both entry points merge it over their defaults, so a per-model quantization choice was already expressible; only the default coupled the two. Users who do want a quantized teacher can pass:

--teacher_model_init_kwargs '{"quantization_config": {"load_in_4bit": true}}'

from_pretrained accepts Union[QuantizationConfigMixin, Dict] for quantization_config (checked against transformers 5.11.0), and teacher_model_init_kwargs is listed in _VALID_DICT_FIELDS on both DistillationConfig and GOLDConfig (trl/experimental/gold/gold_config.py:172), so the JSON string form parses on the command line.

Verification

ruff check and ruff format --check at the CI-pinned 0.13.3, and the pinned doc-builder at --max-len 119, all pass on both files. Both files parse, and no reference to the two deleted scripts remains on the branch.

This is an examples and entry-point change with no test coverage of its own; the behavior it alters is which kwargs reach from_pretrained for the teacher.

---

Note

Medium Risk
Changes default model loading for distillation entrypoints: teachers use more VRAM than before when students are quantized, which may surprise users or cause OOM on tight GPUs.

Overview
Stops applying the student’s quantization_config to the teacher in trl/scripts/distillation.py and examples/gold_chatbot_arena/gold_chatbot_arena.py. With --load_in_4bit (or other ModelConfig quantization flags), only the student stays quantized; the teacher loads in full precision unless overridden.

Comments in both scripts explain that quantizing the teacher weakens the reference distribution for distillation/GOLD, and document opting in via --teacher_model_init_kwargs '{"quantization_config": {...}}'. The distillation CLI comment about student quantization is trimmed now that the teacher no longer carries the shared config by default.

^{Reviewed by Cursor Bugbot for commit 2a5ef8a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[behroozazarkhalili]

Refreshed this branch against main and re-targeted it, because the three files it originally edited no longer exist.

#6820 reorganized examples into per-example folders and #6765 moved the distillation entry point out of examples/scripts/. Merging main produced modify/delete conflicts on examples/scripts/distillation.py and examples/scripts/gkd.py, which I resolved by accepting the deletions. Git's rename detection carried the change into the two surviving call sites on its own:

• trl/scripts/distillation.py
• examples/gold_chatbot_arena/gold_chatbot_arena.py

gkd.py has no surviving equivalent that loads a teacher locally, so that hunk is simply gone.

The bug is unchanged in the relocated GOLD example: line 80 builds one quantization_config, and lines 86 and 97 hand the same object to the student and the teacher.

Two things a reviewer should look at rather than take on trust. In trl/scripts/distillation.py the diff drops one sentence from an existing comment, which stated that the teacher carries quantization through teacher_model_init_kwargs; that described the coupled default this PR removes, so it is no longer true. And teacher_model_init_kwargs already exists on both configs, so this changes a default rather than adding a capability; GOLDConfig lists it in _VALID_DICT_FIELDS at trl/experimental/gold/gold_config.py:172, so the JSON form still parses on the command line.

The branch is now 0 commits behind main. ruff check, ruff format --check at the CI-pinned 0.13.3, and the pinned doc-builder at --max-len 119 all pass on both files. I have updated the title and description to match the new scope.