Proposal
Add a TransformerBridge adapter for Cohere2ForCausalLM (Cohere Command R7B / Command A), the next generation of the already-supported CohereForCausalLM (Command-R v1).
Motivation
Cohere2 is a clean new-generation gap with a distinctive, interp-relevant attention layout: a strict 3:1 interleave of sliding-window attention layers (with RoPE) and full-attention layers (with no positional embedding – NoPE), plus Cohere's logit_scale and GQA. The NoPE-global / RoPE-local split is a clean, interp-relevant target. It lets researchers study how positional information is (and isn't) routed across the two layer types, on a widely-used multilingual family. It is native and low-effort, extend the existing cohere.py (v1) with the interleave (the NoPE handling can borrow SmolLM3's), so very little new machinery.
Verified configs: c4ai-command-r7b-12-2024 (32 layers, hidden 4096, 32/8 GQA, sliding_window=4096, position_embedding_type=rope_gptj, logit_scale=0.25) and c4ai-command-a-03-2025 (64 layers, hidden 12288). One adapter spans Command-R7B (7B, runnable) → Command-A (111B).
Implementation note: the 3:1 interleave is encoded two ways across the family. Command-R7B uses an integer sliding_window_pattern (=4), while command-a-reasoning-08-2025 uses an explicit layer_types list. The adapter must handle both forms. NoPE-on-global-layers is a modeling-code property (RoPE is applied only on sliding layers), not a standalone config flag.
Gap scan (2026-06-25): ~27 models, ~1.09M downloads. Gated (Command license — accessible with HF_TOKEN).
Pitch
Subclass the Cohere v1 block; drive the sliding/full interleave off sliding_window_pattern/layer_types, and suppress RoPE on the full-attention (NoPE) layers.
- Claude Code users can scaffold with
/add-model-support CohereLabs/c4ai-command-r7b-12-2024.
- Register at the four sites listed in contributing.md.
- Verify smallest-first:
trl-internal-testing/tiny-Cohere2ForCausalLM (ungated, CI), then CohereLabs/c4ai-command-r7b-12-2024 (7B; source HF_TOKEN from .env).
Additional context
Checklist
Proposal
Add a TransformerBridge adapter for
Cohere2ForCausalLM(Cohere Command R7B / Command A), the next generation of the already-supportedCohereForCausalLM(Command-R v1).Motivation
Cohere2 is a clean new-generation gap with a distinctive, interp-relevant attention layout: a strict 3:1 interleave of sliding-window attention layers (with RoPE) and full-attention layers (with no positional embedding – NoPE), plus Cohere's
logit_scaleand GQA. The NoPE-global / RoPE-local split is a clean, interp-relevant target. It lets researchers study how positional information is (and isn't) routed across the two layer types, on a widely-used multilingual family. It is native and low-effort, extend the existing cohere.py (v1) with the interleave (the NoPE handling can borrow SmolLM3's), so very little new machinery.Verified configs:
c4ai-command-r7b-12-2024(32 layers, hidden 4096, 32/8 GQA,sliding_window=4096,position_embedding_type=rope_gptj,logit_scale=0.25) andc4ai-command-a-03-2025(64 layers, hidden 12288). One adapter spans Command-R7B (7B, runnable) → Command-A (111B).Implementation note: the 3:1 interleave is encoded two ways across the family. Command-R7B uses an integer
sliding_window_pattern(=4), whilecommand-a-reasoning-08-2025uses an explicitlayer_typeslist. The adapter must handle both forms. NoPE-on-global-layers is a modeling-code property (RoPE is applied only on sliding layers), not a standalone config flag.Gap scan (2026-06-25): ~27 models, ~1.09M downloads. Gated (Command license — accessible with
HF_TOKEN).Pitch
Subclass the Cohere v1 block; drive the sliding/full interleave off
sliding_window_pattern/layer_types, and suppress RoPE on the full-attention (NoPE) layers./add-model-support CohereLabs/c4ai-command-r7b-12-2024.trl-internal-testing/tiny-Cohere2ForCausalLM(ungated, CI), thenCohereLabs/c4ai-command-r7b-12-2024(7B; sourceHF_TOKENfrom.env).Additional context
cohere.py(Command-R v1) adapter.hf_scraperarchitecture-gaps pass (2026-06-25).Checklist