Request for example script / guidance on quantizing Qwen3-30B-A3B

[noforit]

Hi @zzz0906 ,

Thanks for the great repository and continuous support!

I am trying to run quantization on Qwen3-30B-A3B using the latest main branch code, but the quantized model's generation quality drops significantly (it fails to give coherent or correct answers).

Could you please share a working example command or script specifically tailored for Qwen3-30B-A3B.

Any guidance or minimal reproducible example would be greatly appreciated! Thank you for your time.

[zzz0906]

Hi, could you show me how you run it? Do you use any new features like Lloyd-Max, and how do you get the calibrated rotation matrix, and what specific number do you see the drop?

I am busy with rebuttal and serve the larger model. Want to minimize the effort to help you.

Thanks.

[noforit]

I ran Qwen3-30B-A3B with the OSCAR INT2 backend on the main branch. @zzz0906

Repository:
- branch: main
- HEAD: 41ebcdba3db5f0ce1339c3727caea80df575d437
- model: Qwen/Qwen3-30B-A3B
- hardware: 4 x NVIDIA A100 80GB PCIe
- tensor parallel size: 4
- GPUs: 4,5,6,7
- environment: Python 3.12.13, PyTorch 2.9.1+cu128, Transformers 5.3.0, Triton 3.5.1

Calibration and rotations:
- Q/K/V were dumped using the GPQA calibration set.
- dump target: 30,000 tokens
- calibration directory:
  rotation/qwen3-30b-a3b/GPQA/seq30000_prompt121_group128
- 48 transformer layers
- head dimension: 128
- rotation method: qqt_sst
- composition: r_h_pbr
- loaded files:
  - k_rotation_qqt_r_h_pbr.pt
  - v_rotation_sst_r_h_pbr.pt


Server configuration:
- KV cache dtype: INT2
- quantization group size: 128
- mixed KV cache enabled
- BF16 prefix/sink tokens: 64
- BF16 recent tokens: 256
- K clip ratio: 0.96
- V clip ratio: 0.92
- K/V OSCAR rotations enabled
- V rotation absorbed into Qwen3 MoE weights
- max running requests: 32
- max generation tokens: 32768
- temperature: 0.6
- top_p: 0.95
- top_k: 20
- 32 evaluator threads
- no system prompt

The AIME prompt was:

{problem}
Please reason step by step, and put your final answer within \boxed{}.

Results:
- AIME25 has 30 examples.
- With N_REPEATS=5, there were 150 requests.
- 150/150 requests returned HTTP 200.
- Score: 0.07333333333333333
- This corresponds to 11 correct answers out of 150.
- score std: 0.2606828639468953

However, the generated answers were mostly abnormal:
- Most responses entered repetitive loops and continued until the maximum generation length (32768 tokens).
- Many responses repeatedly generated fragments such as `So, So`, `2*2=`, or repeated numeric sequences.
- Most responses did not terminate with a normal `</think>` or final answer.
- Only 25/150 responses contained `</think>`.
- Only 33/150 responses contained a `\boxed{...}` answer.
- Response lengths ranged from approximately 28k to 147k characters, with a median of approximately 72k characters.
- Therefore, the 7.33% score should not be interpreted as normal reasoning accuracy. The main observable behavior was severe repetition/collapse of the generation until `max_tokens` was reached.

If you happen to have some spare time, I would be extremely grateful if you could test/run Qwen3-30B-A3B and share the working script with me!

[zzz0906]

No Problem! I will start a test now! @noforit

[noforit]

No Problem! I will start a test now! @noforit

Thank you so much! Wish you all the best with your rebuttal!