support online quant for quark models

[gbyu-amd]

Motivation

There are cases where we want to online quant the quark models for some specific modules. For example, quant the bf16 attn linear layers to PTPC fp8 for https://huggingface.co/amd/MiniMax-M3-MXFP4, which is already quark quant model.

Technical Details

Test Plan

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[haoyangli0109]

Hi,
@gbyu-amd
It makes sense that bf16 runs successfully, since it doesn’t require dequantization of the weights.
Quark supports multiple quantization formats, and completely opening up the Quark options could pose risks. For example, if we try to perform mxfp4 quantization on a ptpc_fp8 model, problems will arise.

Merging this PR carries some risk, but as long as you confirm there won’t be any misuse, I believe it can be merged.

Based on this PR, we’ll submit another PR to handle online quantization of Quark models in common scenarios next week.

[gbyu-amd]

Hi, @gbyu-amd It makes sense that bf16 runs successfully, since it doesn’t require dequantization of the weights. Quark supports multiple quantization formats, and completely opening up the Quark options could pose risks. For example, if we try to perform mxfp4 quantization on a ptpc_fp8 model, problems will arise.

Merging this PR carries some risk, but as long as you confirm there won’t be any misuse, I believe it can be merged.

Based on this PR, we’ll submit another PR to handle online quantization of Quark models in common scenarios next week.

Thanks @haoyangli0109 , it would be great if you could support more general cases regarding quark models.