Skip to content

feat: Support chat_template_kwargs for OpenAI-compatible models - #1595

Open
mvanhorn wants to merge 1 commit into
simonw:mainfrom
mvanhorn:fix/1491-openai-chat-template-kwargs
Open

feat: Support chat_template_kwargs for OpenAI-compatible models#1595
mvanhorn wants to merge 1 commit into
simonw:mainfrom
mvanhorn:fix/1491-openai-chat-template-kwargs

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Aug 5, 2026

Copy link
Copy Markdown

Summary

Add chat_template_kwargs to the shared Chat Completions option schema as a mapping, with JSON-object normalization for the string form supplied by -o/--option, following the existing logit_bias validation pattern. In _Shared.build_kwargs(), remove that value from the typed OpenAI SDK arguments and place it under the client's custom request-body facility so the serialized request sent to OpenAI-compatible servers contains a top-level chat_template_kwargs object; the shared path will cover synchronous and asynchronous Chat models as well as configured models and llm openai endpoint. Keep the scope explicit to this supported extension instead of changing the base Options policy to accept arbitrary keys.

Validation

  • A saved template whose options.chat_template_kwargs contains enable_thinking: false validates for an OpenAI Chat model and sends that nested object as the top-level chat_template_kwargs property in the HTTP request.
  • A -o chat_template_kwargs '{"enable_thinking": false}' value is normalized to the same mapping, while malformed JSON or a non-object JSON value produces the existing clear option-validation error instead of reaching the provider.
  • Existing standard options such as temperature continue to be sent directly, and omitting chat_template_kwargs does not add an empty custom-body field or otherwise change the request payload.
  • The user-facing OpenAI-compatible models documentation shows how to set the option in a reusable template and makes clear that the nested keys are provider-specific.

Why

Templates currently accept an options: mapping, but each selected model validates that mapping against its Pydantic Options class, so chat_template_kwargs is rejected as an extra input. OpenAI-compatible servers such as llama.cpp accept this request property and use its nested values to control template-specific behavior such as disabling thinking. The OpenAI model implementation needs to expose the option while preserving the project's deliberate validation of model options rather than allowing every unknown key through unchecked.

Fixes #1491

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for additional options (e.g., chat_template_kwargs)?

1 participant