Discussing the possibility to integrate this as PEFT benchmark

[BenjaminBossan]

Hey, thank you for this interesting blog post and paper, and for providing the code. As a PEFT maintainer, I was wondering about the possibility to include this benchmark into the PEFT repo benchmark suite. We already have a benchmark for fine-tuning on MetaMath and for image generation, there is a PR to add a RL benchmark too. I think the benefit of adding such a benchmark to PEFT would be higher visibility and also more exhaustive coverage of many PEFT methods. As you know, OFT is already integrated in PEFT so that would also be covered.

I could see two ways to proceed with an integration:

1. Add a completely new benchmark based on the work in this repo.
2. Extend the existing benchmarks to include the additional metrics you present in the paper. We already have very crude ways of measuring "forgetting" by evaluation the fine-tuned model on wikipedia text for MetaMath and calculating image drift for the image benchmark, but having more detailed metrics would be helpful.

Anyway, let me know if you're interested in cooperating.

[huangyangyi]

Hi @BenjaminBossan, thank you very much for reaching out and for your interest in PEFT-Arena. We would be very happy to cooperate on this.

I think both integration directions you mentioned make sense and could be complementary.

For the benchmark side, one natural path would be to package the core PEFT-Arena protocol into a method_comparison example, similar in spirit to the existing MetaMathQA benchmark. Concretely, we could start with a lightweight text benchmark that includes:

• target-domain finetuning, e.g. math reasoning first;
• target evaluation, e.g. Math500 / AIME / AMC-style math evaluation;
• general-retention evaluation, e.g. IFEval / NQ / BBH or a smaller configurable subset;
• comparable PEFT method configs, including LoRA variants, OFT, IA3, VeRA, DoRA, etc.;
• a simple summary table reporting both target gain and general retention.

For the “forgetting” side, I think it would be useful to make the internal diagnostics available as standalone scripts rather than tying them to a specific benchmark. In our release we already provide lightweight single-checkpoint scripts for CSD and activation-space geometry diagnostics. These could be adapted into PEFT as optional tools where users provide:

• a base model and a finetuned PEFT adapter or merged checkpoint;
• sampled prompts from a target-domain distribution and one or more general-domain distributions;
• one or more modules/layers to probe.

The scripts could then report metrics such as capability-conditioned drift, activation drift, Procrustes residual, Gram distortion, and CKA. This would make the forgetting measurement reusable not only for a new PEFT-Arena-style benchmark, but also for existing PEFT finetuning benchmarks.

One thing to keep in mind is that some parts of our original experiments may need to be revalidated in the PEFT/TRL stack. Our SFT can probably be ported relatively directly, but our target domain evaluations and RLVR/GRPO experiments were implemented with VeRL and vLLM, so reproducing the RL side inside the Hugging Face ecosystem would likely require a PyTorch/TRL-based reimplementation and some sanity checks. We would also be interested in testing whether the proposed forgetting diagnostics are useful on your existing finetuning benchmarks, including the current MetaMath setup.

If this sounds reasonable, I can start by preparing a concrete proposal or draft PR for the PEFT repo, perhaps beginning with:

1. a minimal PEFT-Arena-style text benchmark under method_comparison, and
2. standalone forgetting/geometry diagnostic scripts that can be reused by other benchmarks.

Thanks again for taking a look at our work. I would be very happy to continue the discussion here on GitHub, or by email at yyhuang@link.cuhk.edu.hk.

[BenjaminBossan]

Thanks for your reply and your interest in cooperating.

I believe it would make sense to start working on 2. first. We could integrate these metrics into PEFT: If they are sufficiently general, they could be added to the PEFT package itself, otherwise we provide them to the existing MetaMath benchmark. There, we can augment the existing metrics by adding those new ones and reporting them in the same way. A small caveat is that they should be reasonably fast to run, so that each benchmark can finish in less than ~1 hour.

As for a new benchmark, let's think about it in a second step. I would try to avoid having a benchmark with the same or a very similar dataset as the existing one (MetaMathQA, GSM8K). The different benchmarks should be distinct enough to cover a wide range of possible applications.

Regarding the use of RL framework: It's absolutely not required that we use trl. In fact, I would even prefer not to use any training framework and use vanilla PyTorch instead, but for this task, this will probably require too much extra code. If we use a framework, trl would be the preferred choice because we regularly ensure compatibility with PEFT, but if trl doesn't support the required techniques, we can also try VeRL.

[huangyangyi]

Sounds good to me! I would prepare a PR adding for metric integration first.

[BenjaminBossan]

Fantastic, thank you!