Add GLM-4.7 MoE model support

[sufubao]

python -m lightllm.server.api_server --model_dir /dev/shm/GLM-4.7/ --tp 8 --max_req_total_len 202752

# accuracy check
python ./test/test_api/test_gsmk.py --num-questions 1000 --port 8000
100%|██████████████████████████| 1000/1000 [01:46<00:00,  9.38it/s]
Accuracy: 0.959
Invalid: 0.000
Latency: 106.773 s

# benchmark
============ Serving Benchmark Result ============
Backend:                                 vllm
Traffic request rate:                    inf
Max request concurrency:                 not set
Successful requests:                     1000
Benchmark duration (s):                  75.95
Total input tokens:                      297150
Total input text tokens:                 297150
Total generated tokens:                  194059
Total generated tokens (retokenized):    194043
Request throughput (req/s):              13.17
Input token throughput (tok/s):          3912.50
Output token throughput (tok/s):         2555.12
Peak output token throughput (tok/s):    8134.00
Peak concurrent requests:                1000
Total token throughput (tok/s):          6467.62
Concurrency:                             375.36
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   28508.58
Median E2E Latency (ms):                 28429.38
P90 E2E Latency (ms):                    45289.75
P99 E2E Latency (ms):                    55608.76
---------------Time to First Token----------------
Mean TTFT (ms):                          7648.08
Median TTFT (ms):                        7433.17
P99 TTFT (ms):                           13111.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          349.80
Median TPOT (ms):                        142.88
P99 TPOT (ms):                           3805.27
---------------Inter-Token Latency----------------
Mean ITL (ms):                           108.50
Median ITL (ms):                         55.81
P95 ITL (ms):                            154.64
P99 ITL (ms):                            280.97
Max ITL (ms):                            11435.30
==================================================

# benchmark vs SGLang
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 not set
Successful requests:                     1000
Benchmark duration (s):                  83.18
Total input tokens:                      297150
Total input text tokens:                 297150
Total generated tokens:                  194059
Total generated tokens (retokenized):    193830
Request throughput (req/s):              12.02
Input token throughput (tok/s):          3572.38
Output token throughput (tok/s):         2333.01
Peak output token throughput (tok/s):    7166.00
Peak concurrent requests:                1000
Total token throughput (tok/s):          5905.39
Concurrency:                             388.59
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   32322.99
Median E2E Latency (ms):                 31198.81
P90 E2E Latency (ms):                    50799.95
P99 E2E Latency (ms):                    62104.49
---------------Time to First Token----------------
Mean TTFT (ms):                          9145.00
Median TTFT (ms):                        9131.92
P99 TTFT (ms):                           16137.12
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          444.81
Median TPOT (ms):                        159.32
P99 TPOT (ms):                           5199.22
---------------Inter-Token Latency----------------
Mean ITL (ms):                           120.11
Median ITL (ms):                         65.75
P95 ITL (ms):                            129.58
P99 ITL (ms):                            497.77
Max ITL (ms):                            14352.57
==================================================

[gemini-code-assist]

Summary of Changes

Hello @sufubao, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the necessary infrastructure and implementation to support the GLM-4.7 Mixture-of-Experts (MoE) model within the system. It encompasses the integration of its distinct architectural characteristics, such as specialized attention mechanisms and a dynamic FFN structure, alongside a dedicated Multi-Token Prediction (MTP) variant for enhanced decoding strategies. The changes ensure proper handling of model weights, inference logic, and parallelism strategies for this advanced model.

Highlights

• GLM-4.7 MoE Model Support: Full integration of the GLM-4.7 Mixture-of-Experts (MoE) model architecture, including its unique features.
• Advanced MoE Configuration: Implements 160 routed experts and 1 shared expert with top-8 routing, sigmoid gating, e_score_correction_bias, and a routed_scaling_factor of 2.5.
• Attention Enhancements: Incorporates QK normalization (RMSNorm on Q and K projections) and partial rotary embeddings (0.5 factor) for improved attention mechanisms.
• Hybrid FFN Structure: The first 3 layers utilize dense Feed-Forward Networks (FFN), while subsequent layers employ the MoE FFN structure.
• Multi-Token Prediction (MTP) Variant: Adds support for the MTP variant, designed for speculative decoding, which focuses solely on FFN computations without attention.
• Parallelism Support: Includes logic for both Tensor Parallelism (TP) and Expert Parallelism (EP) modes for MoE layers, and handles overlapped TPSP (Token/Prefill Sequence Parallelism) for MoE.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for the GLM-4.7 MoE model architecture, including its MTP (Multi-Token Prediction) variant for speculative decoding. The implementation covers the specific features of this model, such as QK normalization, partial rotary embeddings, and the unique MoE structure with shared experts. The code is well-structured, adding new modules for the model, its layers, and weights. My review focuses on ensuring robustness, maintainability, and correctness, especially in the context of distributed execution and quantization. I've identified a few areas for improvement, including a potential issue with quantization configuration loading and a risky class reuse that could affect future maintainability. Overall, this is a solid contribution that extends the framework's model support.

[gemini-code-assist]

The weight name kv_b_proj seems incorrect for getting the quantization method for shared FFN experts. This could be a copy-paste error. It should probably be a name related to an FFN projection, like gate_up_proj, to ensure the correct quantization configuration is used. Using the wrong quantization method could lead to incorrect behavior or errors when quantization is enabled.

Suggested change

	kv_b_quant_method = self.quant_cfg.get_quant_method(self.layer_num_, "kv_b_proj")
	weight_scale_suffix = kv_b_quant_method.weight_scale_suffix
	ffn_quant_method = self.quant_cfg.get_quant_method(self.layer_num_, "gate_up_proj")
	weight_scale_suffix = ffn_quant_method.weight_scale_suffix

[gemini-code-assist]

The local import of deep_ep can cause an ImportError if the package is not installed, as this execution path might be taken. It's better to handle this possibility, for example by wrapping it in a try...except block, or by using a global flag set during a guarded import at the top of the file, similar to how it's done in other parts of the codebase (e.g., lightllm/common/fused_moe/grouped_fused_moe_ep.py).

[gemini-code-assist]

Reusing Deepseek3MTPPreLayerInfer directly is a bit risky because its methods are type-hinted to expect Deepseek3MTPPreAndPostLayerWeight, but this model uses Glm4MoeMTPPreAndPostLayerWeight. While it may work now due to duck typing (if the attributes match), it's fragile and can break with future changes. It would be more robust to create a Glm4MoeMTPPreLayerInfer class, possibly inheriting from Deepseek3MTPPreLayerInfer or just copying its logic, to ensure type consistency and better maintainability.