tool_choice="auto" returns tool calls as text with empty tool_calls (only "required" yields structured tool_calls)

[justinchuby]

Summary

When calling the OpenAI-compatible /v1/chat/completions endpoint with tool_choice="auto" (the OpenAI default, and what most agent frameworks send), Foundry Local returns the tool call as raw text inside message.content with an empty tool_calls array and finish_reason="stop". The structured tool_calls are only populated when the caller explicitly sets tool_choice="required".

This breaks standard OpenAI clients and agent frameworks (e.g. anything built on the OpenAI SDK, LangChain, custom agents), because they read message.tool_calls and — per the OpenAI spec — expect tool_choice="auto" to let the model decide and still return structured tool calls when it chooses to call a tool. With Foundry Local they instead receive prose containing an XML/JSON blob that they cannot dispatch.

Reproduction

Model: qwen2.5-coder-7b-instruct-generic-cpu (also reproduces on other tool-capable models), Foundry Local 0.10.0.

Request (only tool_choice differs):

curl -s http://127.0.0.1:<port>/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "qwen2.5-coder-7b-instruct-generic-cpu",
  "messages": [{"role": "user", "content": "What is the weather in Paris? Use the tool."}],
  "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "description": "Get weather for a city",
      "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
  "tool_choice": "auto",
  "temperature": 0
}'

Actual behavior

tool_choice="auto" — tool call leaks into content as text, tool_calls empty:

finish_reason: stop
tool_calls:    []
content:       "```xml\n<tools>\n    {\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\"}}\n</tools>\n```"

tool_choice="required" — structured tool call is returned as expected:

finish_reason: tool_calls
tool_calls:    [{"index": 0, "id": "09xikmhSQ", "type": "function",
                "function": {"name": "get_weather", "arguments": "{\n  \"city\": \"Paris\"\n}"}}]

Expected behavior

With tool_choice="auto", when the model decides to call a tool, Foundry Local should parse the model's tool-call output and populate message.tool_calls with finish_reason="tool_calls" — the same as tool_choice="required" — instead of returning the call as raw text. auto should differ from required only in whether a tool must be called, not in how a chosen call is surfaced.

Impact

• Any OpenAI-compatible agent/framework that sends the default tool_choice="auto" cannot use Foundry Local tool calling without a custom text-scraping workaround.
• Forcing tool_choice="required" is not a general fix, because it prevents the model from answering directly when no tool is needed.

Environment

• Foundry Local: 0.10.0
• Endpoint: OpenAI-compatible /v1/chat/completions
• Models: qwen2.5-coder-7b-instruct-generic-cpu (reproduces broadly across tool-capable models)
• OS: Linux

Question

Is the auto → text vs required → structured behavior intentional, or a gap in the server-side tool-call parsing? Ideally auto would run the same tool-call extraction that required already does and emit structured tool_calls.

[justinchuby]

@bmehta001

[rymikula]

Corroborating on Foundry Local 0.8.119 / C# SDK 1.2.3 with Phi-4-mini-instruct-generic-gpu:5 (catalog lists it as chat, tools). For Phi it goes one level deeper: under tool_choice: "auto" the request's tools are never rendered into the prompt at all (identical prompt_tokens with and without 3 tools attached). If we inject the tool schemas into the system prompt ourselves, the model does call correctly, but as a bare JSON array [{"name": …, "arguments": {…}}] in content, with empty tool_calls and finish_reason: "stop", exactly as reported here. tool_choice: "required" returns proper structured tool_calls, but forcing a call every turn isn't viable for an agent loop.

Note for the fix: marker-token detection (#838, onnxruntime-genai#2215) won't catch this case, because under auto Phi never sees the tools, so its reply has no marker tokens. Phi needs both halves: render tools into the chat template, and lift the reply into tool_calls under auto like required already does.