How to input audio to Gemma 4 E4B?

[burgil]

Hello, according to the Gemma 4 model card the model supports audio inputs, I wonder is there a way to enable that?

Source: https://ai.google.dev/gemma/docs/core/model_card_4#6_audio

Command used:

llama-b8639-bin-win-cuda-13.1-x64>llama-server -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_K_M -c 24676 -ngl 99 --flash-attn on --jinja

Related search: https://github.com/ggml-org/llama.cpp/discussions?discussions_q=Gemma+4+audio

[SmartestWashingMachine]

llama-cpp does not support gemma 4 audio as of the time of this post

see: #21309

[burgil]

But.. But.. I'm so close! plz help

srv    operator(): got exception: {"error":{"code":500,"message":"audio input is not supported - hint: if this is unexpected, you may need to provide the mmproj","type":"server_error"}}

Edit Example Audio: https://ai.google.dev/gemma/docs/audio/roses-are.wav

[SmartestWashingMachine]

Currently llama-cpp will not parse audio with Gemma, but this may change in the future. also for future reference, llama-cpp requires the mmproj to be loaded for audio to work, so download the mmproj and then your command should look like this:

EDIT: I'm not sure how mmproj interacts with -hf flag, so this may be redundant.

llama-server -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_K_M -c 24676 -ngl 99 --flash-attn on --jinja --mmproj mmproj-BF16.gguf

And doing this in code would look like so:

import openai
import base64

client = openai.OpenAI(
    base_url="http://localhost:62322/v1", 
    api_key="i-love-gemma"
)

# Read and encode the WAV file
with open(r"sample.wav", "rb") as f:
    encoded_audio = base64.b64encode(f.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma_:(",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Please transcribe this audio into text"},
                {
                    "type": "input_audio", 
                    "input_audio": {
                        "data": encoded_audio, 
                        "format": "wav" # can change as needed.
                    }
                }
            ],
        }
    ],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

But again, audio input for Gemma4 in llama-cpp is not supported yet. For now a different library would be necessary.

[burgil]

Thank you very much, You were right it's not support no matter how hard I try to force it to support it, in the model card they made it look very easy:

C:\Users\myuser\Downloads\llama-b8639-bin-win-cuda-13.1-x64>llama-server -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_K_M -c 24676 -ngl 99 --flash-attn on --jinja --mmproj C:\Users\myuser\.cache\huggingface\hub\models--ggml-org--gemma-4-E4B-it-GGUF\snapshots\6b352c53e1d2e4bb974d9f8cafcf85887c224219\mmproj-gemma-4-e4b-it-f16.gguf

srv    operator(): got exception: {"error":{"code":500,"message":"audio input is not supported - hint: if this is unexpected, you may need to provide the mmproj","type":"server_error"}}
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 500
srv    operator(): operator(): cleaning up before exit...
llama_memory_breakdown_print: | memory breakdown [MiB]  | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - CUDA0 (RTX 3060 Ti) |  8191 = 2788 + (3873 =  2868 +     488 +     517) +        1530 |
llama_memory_breakdown_print: |   - Host                |                 2809 =  2730 +       0 +      79                |

I'll try the python code, and I guess a different library will be needed indeed

[burgil]

Figured it out, finally: #21325 (comment)