Skip to content

Comments

feat: add vLLM 0.16 mistral option parity for exllamav3#415

Open
lesj0610 wants to merge 13 commits intotheroyallab:mainfrom
lesj0610:feat/vllm-mistral-parity
Open

feat: add vLLM 0.16 mistral option parity for exllamav3#415
lesj0610 wants to merge 13 commits intotheroyallab:mainfrom
lesj0610:feat/vllm-mistral-parity

Conversation

@lesj0610
Copy link

Summary

  • implement tool_call_parser=mistral for both Mistral formats
    • legacy JSON list: [TOOL_CALLS] [{"name":...,"arguments":...}]
    • v11+ segmented format: [TOOL_CALLS]name{...}[TOOL_CALLS]...
  • wire parser dispatch and native-generation behavior for mistral parser key
  • add tokenizer_mode model config/plumbing and expose it in model metadata
  • apply mistral-safe tool call id normalization/truncation in chat formatting path
  • improve mistral reasoning parser token handling with mistral_common special tokens fallback
  • keep non-mistral models on existing/default paths (fallback-safe behavior)

Files / Areas

  • endpoints/OAI/utils/tools.py
  • endpoints/OAI/utils/chat_completion.py
  • endpoints/OAI/utils/parser_options.py
  • common/config_models.py
  • endpoints/core/types/model.py
  • backends/exllamav3/model.py
  • backends/exllamav2/model.py
  • docs + tests updates

Validation

  • PYTHONPATH=. pytest -q tests/tool_parser_test.py tests/parser_options_test.py tests/mistral_reasoning_parser_test.py tests/exaone4_reasoning_parser_test.py tests/wheel_test.py
  • result: 34 passed
  • note: full pytest -q in this environment still fails in tests/model_test.py due missing exllamav2 package (ModuleNotFoundError), unrelated to this change set.

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants