[FEATURE] Voice interaction with agents via Gemini Multimodal Live API

[braghettos]

Prerequisites

• I have searched the existing issues to make sure this feature has not been requested before
• I agree to follow the Code of Conduct

📝 Feature Summary

Add voice interaction support to kagent agents via the Gemini Multimodal Live API, allowing users to speak to agents and receive spoken responses through native audio processing with function calling bridging voice to the A2A protocol.

❓ Problem Statement / Motivation

Currently, kagent agents can only be interacted with via text (CLI, kagent UI, Slack integration). Voice interaction would:

• Enable hands-free operation for SREs during incident response (e.g., "What pods are crashing in production?")
• Make the platform more accessible to non-technical stakeholders who prefer natural conversation over typing commands
• Reduce friction for quick queries — speaking is faster than typing for many operations

The Gemini Multimodal Live API makes this feasible without building a separate STT/TTS pipeline, since it handles audio input/output natively while supporting function calling to bridge into the A2A protocol.

💡 Proposed Solution

A lightweight voice interface that connects Gemini Live API to any kagent agent via A2A:

Architecture:

Browser (mic/speaker)
    │
    ├── WebSocket ──→ Gemini Live API (native audio in/out)
    │                     │
    │                     └── tool_call: "send_to_agent"
    │                              │
    └── JS handler ──→ Backend proxy ──→ kagent A2A endpoint

Components:

1. Frontend — Browser-based voice UI using Web Audio API for mic/speaker, WebSocket to Gemini Live API
2. Backend — Lightweight proxy (FastAPI/Go) that forwards tool call requests from the browser to the kagent A2A endpoint, avoiding CORS and keeping API keys server-side
3. Gemini Live session — Configured with a send_to_agent function declaration so Gemini routes platform/K8s requests through A2A while handling greetings and simple questions directly

Key details:

• Audio: 16kHz PCM input, 24kHz PCM output, with input/output transcription for visual display
• Function calling: Gemini Live supports synchronous tool use — when the user asks for a K8s action, Gemini calls the tool, waits for the A2A response, and speaks the result naturally
• Model: gemini-2.5-flash-live-preview (low latency, supports audio + tools)
• Could be deployed standalone or integrated into kagent UI as a voice mode toggle

🔄 Alternatives Considered

• Browser Web Speech API — Free but lower quality STT/TTS, requires a separate text-to-A2A bridge, no native function calling
• Google Cloud Speech + TTS — High quality but adds two separate services to manage, more latency from STT→text→LLM→text→TTS pipeline
• OpenAI Realtime API — Good quality but adds an OpenAI dependency when kagent already uses Gemini

Gemini Live is the best fit because it eliminates the STT/TTS pipeline entirely and natively supports function calling to bridge into A2A.

🎯 Affected Service(s)

UI Service

📚 Additional Context

• Gemini Multimodal Live API docs
• Live API Tool Use
• google-gemini/live-api-web-console — React starter from Google
• We have a working prototype in the Krateo autopilot-kagent repo (autopilot/voice-ui/) that demonstrates this approach with a FastAPI backend + vanilla JS frontend

🙋 Are you willing to contribute?

• I would be willing to submit a PR to implement this feature

[m13v]

built a voice-first macOS agent and the biggest lesson was that the STT-to-action latency is what makes or breaks the UX. users tolerate ~500ms total from speech end to first visible response, beyond that it feels broken.

the Gemini Multimodal Live API approach is smart because it avoids the STT -> text -> LLM -> TTS pipeline overhead. but for the SRE use case you described, one thing to consider: incident response commands tend to be short and domain-specific ("scale deployment X to 5 replicas"), which means you can get really good accuracy with a smaller local model like WhisperKit running on-device for the STT step. the tradeoff is you lose Gemini's native function calling, but you gain offline capability and sub-100ms transcription latency.

we found that hybrid works best - local model for quick commands, cloud API for complex multi-turn conversations. the mode switch can be automatic based on utterance length.

[m13v]

our voice transcription service that handles the local WhisperKit pipeline with streaming audio input: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

and the tool execution layer that bridges voice commands to actual function calls: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift

[braghettos]

Thanks @m13v for the insights on latency — the 500ms threshold is spot on.

After investigating the kagent codebase, I can confirm that kagent UI already has a voice input button (via useSpeechRecognition hook in /ui/src/hooks/useSpeechRecognition.ts). However, the current implementation:

1. Speech-to-text only — it uses the browser's native Web Speech API (SpeechRecognition / webkitSpeechRecognition) to transcribe voice into text and paste it into the chat input. No audio output.
2. English-only in practice — the hook accepts a language parameter but ChatInterface.tsx doesn't pass one, so it defaults to en-US.
3. No tool calling / function integration — it's a convenience feature for text input, not a voice agent.

Proposal: Evolve the existing mic button

Rather than building a separate voice app, I'd suggest evolving the existing kagent UI mic button to integrate a voice agent that fully leverages the kagent architecture:

1. Use kagent ModelConfig for the voice model — instead of hardcoding a specific provider, let the user reference an existing ModelConfig that points to any model supporting real-time audio (Gemini Live, OpenAI Realtime, etc.). This keeps the voice layer model-agnostic, consistent with how kagent agents already work. The UI would read the ModelConfig to determine the WebSocket endpoint, auth, and protocol.
2. Declare kagent A2A as a tool — the voice session declares a send_to_agent function. When the user speaks a K8s command, the model calls the function, which proxies to the agent's A2A endpoint. The result is spoken back naturally.
3. Multilanguage out of the box — models like Gemini Live support 24 languages natively with auto-detection. No configuration needed.
4. Context-aware follow-ups — after each response, generate suggested follow-up questions as clickable chips below the chat.

This means a user could configure:

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: voice-model
spec:
  model: gemini-3.1-flash-live-preview
  provider: GeminiLive
  apiKeySecret: gemini-api-key
  apiKeySecretKey: apiKey

And the UI mic button would use whatever ModelConfig is configured for voice, just like agents use ModelConfig for their LLM. If tomorrow OpenAI ships a better realtime model, users just swap the ModelConfig — no UI changes needed.

We have a working prototype in our Krateo autopilot voice-ui that demonstrates the full flow (currently Gemini-specific, but the architecture maps cleanly to a ModelConfig-driven approach).

The key insight: the voice layer is thin — it's just a "voice frontend" to the existing A2A protocol. All agent logic stays in kagent.

@m13v great point about domain-specific commands. The function calling approach handles this well — the model learns the agent's capabilities from the tool declaration and routes accordingly. Short commands like "scale X to 5" get routed to the tool immediately without unnecessary conversation.