feat: Support chat_template_kwargs for OpenAI-compatible models - #1595
Open
mvanhorn wants to merge 1 commit into
Open
feat: Support chat_template_kwargs for OpenAI-compatible models#1595mvanhorn wants to merge 1 commit into
mvanhorn wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
chat_template_kwargsto the shared Chat Completions option schema as a mapping, with JSON-object normalization for the string form supplied by-o/--option, following the existinglogit_biasvalidation pattern. In_Shared.build_kwargs(), remove that value from the typed OpenAI SDK arguments and place it under the client's custom request-body facility so the serialized request sent to OpenAI-compatible servers contains a top-levelchat_template_kwargsobject; the shared path will cover synchronous and asynchronousChatmodels as well as configured models andllm openai endpoint. Keep the scope explicit to this supported extension instead of changing the baseOptionspolicy to accept arbitrary keys.Validation
options.chat_template_kwargscontainsenable_thinking: falsevalidates for an OpenAI Chat model and sends that nested object as the top-levelchat_template_kwargsproperty in the HTTP request.-o chat_template_kwargs '{"enable_thinking": false}'value is normalized to the same mapping, while malformed JSON or a non-object JSON value produces the existing clear option-validation error instead of reaching the provider.temperaturecontinue to be sent directly, and omittingchat_template_kwargsdoes not add an empty custom-body field or otherwise change the request payload.Why
Templates currently accept an
options:mapping, but each selected model validates that mapping against its PydanticOptionsclass, sochat_template_kwargsis rejected as an extra input. OpenAI-compatible servers such as llama.cpp accept this request property and use its nested values to control template-specific behavior such as disabling thinking. The OpenAI model implementation needs to expose the option while preserving the project's deliberate validation of model options rather than allowing every unknown key through unchecked.Fixes #1491