Add qwen3.6 recipe - #561
Open
apsonawane wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “builtin” Olive recipe for exporting, optimizing, running inference, and evaluating the Qwen3.6-27B vision-language model using ONNX Runtime GenAI.
Changes:
- Introduces end-to-end scripts for export/optimization (
optimize.py), inference (inference.py), and AI2D evaluation (eval.py). - Adds Olive pipeline configs for CPU INT4 and CUDA FP16/INT4 workflows (embedding/vision/text submodels).
- Includes supporting documentation, dependency list, ignore rules, and embedded model code needed for export.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| Qwen-Qwen3.6-27B/LICENSE | Adds Apache 2.0 license text for the recipe/model artifacts. |
| Qwen-Qwen3.6-27B/builtin/.gitignore | Ignores generated model outputs, bytecode, and Olive cache. |
| Qwen-Qwen3.6-27B/builtin/README.md | Documents hardware requirements and step-by-step usage. |
| Qwen-Qwen3.6-27B/builtin/requirements.txt | Declares Python dependencies needed for export/eval/inference. |
| Qwen-Qwen3.6-27B/builtin/info.yml | Registers the recipe entrypoint and supported EPs/devices. |
| Qwen-Qwen3.6-27B/builtin/optimize.py | Runs Olive pipelines and patches GenAI + processor + tokenizer configs. |
| Qwen-Qwen3.6-27B/builtin/inference.py | Provides ONNX Runtime GenAI inference, interactive mode, and benchmarking. |
| Qwen-Qwen3.6-27B/builtin/eval.py | Evaluates ONNX vs PyTorch accuracy/latency on AI2D dataset. |
| Qwen-Qwen3.6-27B/builtin/user_script.py | Loads HF weights into a custom model shell for embedding/vision export. |
| Qwen-Qwen3.6-27B/builtin/cpu_and_mobile/embedding.json | CPU INT4 Olive pipeline for embedding submodel export/quantization. |
| Qwen-Qwen3.6-27B/builtin/cpu_and_mobile/vision.json | CPU INT4 Olive pipeline for vision submodel export/quantization. |
| Qwen-Qwen3.6-27B/builtin/cpu_and_mobile/text.json | CPU INT4 ModelBuilder pipeline for text decoder. |
| Qwen-Qwen3.6-27B/builtin/cuda/embedding.json | CUDA FP16 Olive pipeline for embedding submodel export/optimization. |
| Qwen-Qwen3.6-27B/builtin/cuda/vision.json | CUDA FP16 Olive pipeline for vision submodel export/optimization. |
| Qwen-Qwen3.6-27B/builtin/cuda/text.json | CUDA INT4 ModelBuilder pipeline for text decoder. |
| Qwen-Qwen3.6-27B/builtin/codes/init.py | Initializes the bundled modeling package. |
| Qwen-Qwen3.6-27B/builtin/codes/modeling_qwen3_5.py | Adds the custom model implementation used during export. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+47
to
+51
| # Buffers such as `visual.rotary_pos_emb.inv_freq` are recomputed at model | ||
| # init from config and are commonly absent from the checkpoint — treat any | ||
| # remaining missing keys as a warning rather than a hard failure. | ||
| if result.missing_keys: | ||
| print(f"Warning: {len(result.missing_keys)} missing keys (first 10: {result.missing_keys[:10]})") |
|
|
||
| This example demonstrates how to convert [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) vision-language model to ONNX format using Olive and run inference with ONNX Runtime GenAI. | ||
|
|
||
| Qwen3.5 is a hybrid architecture combining GatedDeltaNet linear attention layers with standard full attention layers. The pipeline exports three sub-models (vision encoder, text embedding, text decoder) and applies graph optimizations. For CPU, all three sub-models are quantized to INT4. For CUDA, the vision encoder and embedding use FP16 while the text decoder uses INT4. |
Comment on lines
+127
to
+129
| Qwen3.5's tokenizer uses Unicode property escapes (\\p{L}, \\p{N}) in its | ||
| Split pre-tokenizer, which aren't supported by std::regex in onnxruntime-genai. | ||
| Remove the Split and keep only ByteLevel with use_regex=True. |
| # Use 1/2/3/4 instead of A/B/C/D to avoid collisions with AI2D region labels. | ||
| NUMBERS = ["1", "2", "3", "4"] | ||
|
|
||
| # System prompt with /no_think to suppress verbose reasoning in Qwen3.5. |
Comment on lines
+195
to
+202
| def _gpu_memory_mb(): | ||
| """Return (current, peak) GPU memory in MB, or (0, 0) if unavailable.""" | ||
| try: | ||
| import torch | ||
| if torch.cuda.is_available(): | ||
| cur = torch.cuda.memory_allocated() / (1024 ** 3) | ||
| peak = torch.cuda.max_memory_allocated() / (1024 ** 3) | ||
| return cur, peak |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a complete ONNX Runtime GenAI example for the Qwen3.6-27B vision-language model, including export, optimization, inference, and evaluation scripts and configuration. The changes add all necessary pipeline configurations for both CPU (INT4 quantized) and CUDA (FP16/INT4 hybrid) builds, a comprehensive README, and supporting files for model conversion and deployment.
The most important changes are:
Documentation and Usage Instructions
README.mdinbuiltin/explaining hardware requirements, installation, model export and optimization steps, inference commands, evaluation procedure, and benchmark results for Qwen3.6-27B ONNX Runtime GenAI workflows.Model Export and Optimization Pipelines
cpu_and_mobile/embedding.json,cpu_and_mobile/vision.json,cpu_and_mobile/text.json: Define export and optimization passes for embedding, vision, and text sub-models, including INT4 quantization for CPU. [1] [2] [3]cuda/embedding.json,cuda/vision.json,cuda/text.json: Define export and optimization passes for CUDA, using FP16 for vision/embedding and INT4 for text decoder. [1] [2] [3]Project Structure and Supporting Files
.gitignoreinbuiltin/to exclude model artifacts, Python bytecode, and Olive cache directories.LICENSEfile for the Qwen3.6-27B model, specifying the Apache License 2.0 terms.These changes provide a ready-to-use, reproducible workflow for exporting, optimizing, and running the Qwen3.6-27B model with ONNX Runtime GenAI on both CPU and GPU platforms.