Structured Output problem

[NovaH00]

When I use the OpenAI Python SDK with the llama-server endpoint, I couldn't get the response_format to work!

# llm.py
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI(base_url="http://localhost:8080", api_key="unused")

class CalendarEvent(BaseModel):
    name: str
    date: str
    participants: list[str]

response = client.chat.completions.parse(
    model="gpt-oss-20b",
    messages=[
        {
            "role": "system", 
            "content": "Extract the event information."
        },
        {
            "role": "user",
            "content": "Alice and Bob are going to a science fair on Friday.",
        },
    ],
    response_format=CalendarEvent,
)

print(response)

$ uv run llm.py
...
openai.InternalServerError: Error code: 500 - {'error': {'code': 500, 'message': 'Failed to parse input at pos 610: <|start|>assistant<|channel|>final<|message|>**Event Information**\n\n- **Event**: Science Fair  \n- **Date**: Friday  \n- **Participants**: Alice, Bob', 'type': 'server_error'}}

I found some issues related to this problem and it seem like they all fixed but I don't really know why it's not working

Also I intercepted the payload send by the SDK

{
  "messages": [
    {
      "role": "system",
      "content": "Extract the event information."
    },
    {
      "role": "user",
      "content": "Alice and Bob are going to a science fair on Friday."
    }
  ],
  "model": "gpt-oss-20b",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "schema": {
        "properties": {
          "name": {
            "title": "Name",
            "type": "string"
          },
          "date": {
            "title": "Date",
            "type": "string"
          },
          "participants": {
            "items": {
              "type": "string"
            },
            "title": "Participants",
            "type": "array"
          }
        },
        "required": [
          "name",
          "date",
          "participants"
        ],
        "title": "CalendarEvent",
        "type": "object",
        "additionalProperties": false
      },
      "name": "CalendarEvent",
      "strict": true
    }
  },
  "stream": false
}

Some issue mentioning the fix is to use json_object rather than json_schema but I tried using curl with that and got no luck.

[leonardigabriele]

I am having the same issue using Qwen/Qwen3-VL-8B-Instruct-GGUF:Q4_K_M
The extraction was working perfectly in a previous version of llama.cpp
Can you tell how to make structured output work? I tried json_schema, passing pydantic model and using responses api but no luck.

[NovaH00]

Here's the workaround: #21228

Update the function in llama.cpp/common/chat-auto-parser-generator.cpp (you can just replace the current function to this snippet)

common_chat_params peg_generator::generate_parser(const common_chat_template &    tmpl,
                                                  const struct generation_params & inputs,
                                                  const autoparser &              autoparser) {
    // Create a mutable copy to resolve refs before building the parser
    struct generation_params inputs_resolved = inputs;
    if (inputs_resolved.json_schema.is_object() && !inputs_resolved.json_schema.empty()) {
        // Inline $defs into $ref locations
        json schema_copy = inputs_resolved.json_schema;
        if (schema_copy.contains("$defs")) {
            json defs = schema_copy["$defs"];
            // First resolve nested $refs within $defs
            std::function<void(json&)> resolve_refs_inline = [&](json& n) {
                if (n.is_array()) {
                    for (auto &x : n) resolve_refs_inline(x);
                } else if (n.is_object()) {
                    if (n.contains("$ref") && n["$ref"].is_string()) {
                        std::string ref = n["$ref"].get<std::string>();
                        if (ref.find("#/$defs/") == 0) {
                            std::string def_name = ref.substr(8);
                            if (defs.contains(def_name)) {
                                n = defs[def_name];
                            }
                        }
                    } else {
                        for (const auto &kv : n.items()) resolve_refs_inline(kv.value());
                    }
                }
            };
            // Resolve within $defs first
            for (const auto &kv : defs.items()) {
                resolve_refs_inline(kv.value());
            }
            // Then resolve in the rest of the schema
            resolve_refs_inline(schema_copy);
            schema_copy.erase("$defs");
        }
        inputs_resolved.json_schema = schema_copy;
    }

    // Create the result structure
    common_chat_params data;
    data.prompt           = common_chat_template_direct_apply(tmpl, inputs_resolved);
    data.format           = COMMON_CHAT_FORMAT_PEG_NATIVE;
    data.preserved_tokens = autoparser.preserved_tokens;

    auto parser = autoparser.build_parser(inputs_resolved);
    data.parser = parser.save();

    // Build grammar if tools are present
    bool has_tools =
        autoparser.tools.format.mode != tool_format::NONE && inputs_resolved.tools.is_array() && !inputs_resolved.tools.empty();
    std::string trigger_marker = !autoparser.tools.format.section_start.empty() ? autoparser.tools.format.section_start :
                                                                                  autoparser.tools.format.per_call_start;

    bool has_response_format = !inputs_resolved.json_schema.empty() && inputs_resolved.json_schema.is_object();
    bool include_grammar = has_response_format || (has_tools &&
            ((inputs_resolved.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO && !trigger_marker.empty()) ||
              inputs_resolved.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));

    if (include_grammar) {
        data.grammar_lazy = !has_response_format && inputs_resolved.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
            foreach_function(inputs_resolved.tools, [&](const json & tool) {
                const auto & function = tool.at("function");
                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
                builder.resolve_refs(schema);
            });
            parser.build_grammar(builder, data.grammar_lazy);
        });

        // Set grammar triggers based on tool section markers (fall back to per-call markers)
        if (data.grammar_lazy) {
            data.grammar_triggers = {
                { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, trigger_marker }
            };
        }
    }

    return data;
}

My diff workaround in the issue page (#21228) require resolve some conflicts. The snippet above is the code that has been resolved.

Also this workaround stop working for Gemma 4 models after this commit (b863507). If you want for Gemma 4 to work, use llama.cpp build older than the commit AND use the implementation in the github issue page I provided!

[NovaH00]

The problem has been solved as per https://github.com/ggml-org/llama.cpp/releases/tag/b8756 release