Skip to content

feat: add Motif-3 model architecture support - #1

Open
ajaxdude wants to merge 2 commits into
Nathanw1014:strix-halo-vulkanfrom
ajaxdude:motif3-support
Open

feat: add Motif-3 model architecture support#1
ajaxdude wants to merge 2 commits into
Nathanw1014:strix-halo-vulkanfrom
ajaxdude:motif3-support

Conversation

@ajaxdude

@ajaxdude ajaxdude commented Aug 13, 2026

Copy link
Copy Markdown

Summary

Add support for the Motif-3 model architecture (314B.A13B MoE) from AesSedai/Motif-3-GGUF.

Architecture features

  • GDLA (Grouped Differential Latent Attention): Attention variant with per-head lambda projections and latent KV cache (MLA-style)
  • Grouped PolyNorm MoE: PolyNorm activation with grouped expert routing and shared expert fallback
  • mHC (Manifold-constrained Hyper-Connections): Residual stream expansion with Sinkhorn-normalized gating tensors (h_pre, h_post, h_res)
  • Interleaved SWA + YaRN: Sliding window attention with YaRN rope scaling

Changes

File Change
llama-arch.{h,cpp} MOTIF3 arch enum, 8 KV fields, 14 tensor fields with info
llama-model.{h,cpp} 26 layer tensors, model constructor mapping, rope type
llama-hparams.h PolyNorm eps/out_scale/clamp, mHC mult/iters/post_coeff, GDLA MLA dims
llama-graph.cpp MLA KV cache integration (K-only base cache), buffer null checks
llama-kv-cache-iswa.{h,cpp} SWA subcache hparams override to disable MLA on SWA layers
llama-context.cpp Graph node allocation ceiling for motif3
llama-quant.cpp Motif-3 tensor quantization exclusions (PolyNorm, mHC, GDLA weights)
llama-vocab.cpp <|endofturn|> end-of-turn token
gguf-py/gguf/constants.py Enum, tensor names, allowed/skip tensor lists
conversion/__init__.py MotifForCausalLMmotif3 mapping
conversion/motif3.py PyTorch → GGUF converter (373 lines)
src/models/motif3.cpp Full architecture implementation (907 lines)
src/models/models.h llama_model_motif3 struct declaration

Based on

Upstream PR: ggml-org/llama.cpp#26298

Testing

  • Verified against upstream PR model: Add support for Motif 3 Beta  ggml-org/llama.cpp#26298: all added lines present; new files conversion/motif3.py and src/models/motif3.cpp byte-identical to upstream
  • cmake -B build -G Ninja + ninja llama: compiles clean (CPU build)
  • cmake -B build-vk -G Ninja -DGGML_VULKAN=ON + ninja llama: compiles clean (Vulkan build; libggml-vulkan.so built)
  • Converter unit-tested: all tensor mappings, wkv_b MLA split, mHC alpha grouping, and 33 arch-specific GGUF parameters verified with mocked tensors/writer
  • Picked up automatically via file(GLOB LLAMA_MODELS_SOURCES "models/*.cpp") in src/CMakeLists.txt
  • Full inference test not run: smallest quant (IQ2_S) is ~108 GB

Add support for the Motif-3 model architecture (314B.A13B MoE) from
https://huggingface.co/AesSedai/Motif-3-GGUF.

Changes:
- llama-arch.{h,cpp}: MOTIF3 arch enum, 8 KV fields, 14 tensor fields
- llama-model.{h,cpp}: Layer tensors, model mapping, rope type
- llama-hparams.h: PolyNorm, mHC, GDLA/MLA hyperparameters
- llama-graph.cpp: MLA KV cache integration, buffer checks
- llama-kv-cache-iswa.{h,cpp}: SWA subcache hparams override for MLA
- llama-context.cpp: Graph node allocation for motif3
- llama-quant.cpp: Motif-3 tensor quantization exclusions
- llama-vocab.cpp: <|endofturn|> end-of-turn token
- gguf-py/gguf/constants.py: Enum, names, allowed/skip tensor lists
- conversion/__init__.py: MotifForCausalLM -> motif3 mapping
- conversion/motif3.py: PyTorch -> GGUF converter
- src/models/motif3.cpp: GDLA attention, PolyNorm MoE, mHC, interleaved SWA+Yarn
- src/models/models.h: llama_model_motif3 declaration

Based on upstream PR ggml-org#26298
- remove duplicated LLM_TENSOR_NAMES entries from LLM_TENSOR_INFOS array
- fix llama_model_mapping: restore nanbeige return, remove stray MOTIF3 case
- fix llama_model_rope_type: remove misplaced motif3 return and stray case
- remove dangling endofturn check in llama-vocab.cpp
- restore missing closing brace in motif3.cpp
- match upstream comment wording in kv-cache-iswa.h and quant.cpp
- add MOTIF3 MODEL_TENSORS list to gguf constants
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant