docs(proposals): How? for Google Vertex AI translation (#779) - #784
docs(proposals): How? for Google Vertex AI translation (#779)#784mkoushni wants to merge 2 commits into
Conversation
…oxy#779) Adds the implementation design for the Google Vertex AI (Gemini) sub-task split out of praxis-proxy#762, building on the accepted What?/Why? in 00762_api-translation.md. Signed-off-by: mkoushni <mkoushni@redhat.com>
|
Proposal validation failed:
|
|
LGTM |
Signed-off-by: mkoushni <mkoushni@redhat.com>
|
the example chains use |
aslakknutsen
left a comment
aslakknutsen
left a comment
There was a problem hiding this comment.
API-spec review of the How? mapping tables. Six inline comments on streaming, tools, safety 200s, errors, fail-closed request fields, and the Chat Completions-only consumer surface.
| that bidirectionally rewrites OpenAI Chat Completions-shaped traffic to and | ||
| from Vertex AI's Gemini `generateContent` / `streamGenerateContent` APIs, | ||
| including forcing genuine SSE framing on the streaming path (`?alt=sse`) so | ||
| it can share the SSE decoder family with Azure/Cohere rather than needing a | ||
| bespoke transport, per 00762's explicit resolution of this point. This is | ||
| the Vertex-scoped instance of the general translation stage already accepted | ||
| in [docs/proposals/00762_api-translation.md](00762_api-translation.md); this | ||
| proposal adds the **How?** for | ||
| [praxis-proxy/ai#779](https://github.com/praxis-proxy/ai/issues/779) only. | ||
|
|
||
| ### Goals | ||
|
|
||
| - Translate OpenAI Chat Completions requests to Gemini `generateContent` | ||
| request bodies, and translate Gemini responses (including errors) back to | ||
| OpenAI Chat Completions shape. |
There was a problem hiding this comment.
This locks the consumer contract to Chat Completions.
Praxis already has Responses ↔ Chat Completions (responses_to_chat_completions / apis/src/openai/translation). Either state that /v1/responses is out of scope for #779, or name the composition hop. As written, a Responses client has no documented path onto this Vertex chain. The e2e later (POST /v1/chat/completions, ~L409) only covers one of the two OpenAI surfaces this repo actually serves.
| - OpenAI `messages[]` (role `system`/`user`/`assistant`/`tool`) become | ||
| Gemini `contents[]` with role `user`/`model` (`assistant` → `model`; | ||
| `system` messages are hoisted into the top-level `systemInstruction` | ||
| field, the same hoisting pattern `anthropic_to_openai` already applies | ||
| for Anthropic's `system` field, just targeting a differently-named Gemini | ||
| field); each message's text becomes one `parts: [{text: ...}]` entry. | ||
| - OpenAI `tool_calls`/`tool`-role messages become Gemini `functionCall`/ | ||
| `functionResponse` parts; OpenAI `tools[]` (JSON-schema function defs) | ||
| become `tools: [{functionDeclarations: [...]}]`. | ||
| - `temperature`/`top_p`/`max_tokens`/`stop` map to Gemini's | ||
| `generationConfig {temperature, topP, maxOutputTokens, stopSequences}`. | ||
|
|
||
| Malformed or untranslatable input (no `messages`, unsupported field shape) | ||
| is rejected `400` before ever reaching Vertex — the same fail-closed rule | ||
| 00762 requires for every provider, and the same rule `bedrock_translate` | ||
| and `anthropic_to_openai` already apply. |
There was a problem hiding this comment.
Fail-closed is claimed here, but the mapped field set is not enumerated.
Listed: messages, tools, temperature / top_p / max_tokens / stop. Not decided (400 vs drop vs leak): tool_choice, parallel_tool_calls, response_format, n, logprobs, seed, max_completion_tokens, multimodal image_url parts, and Gemini thought: true parts.
Roles also need a rule: Gemini Content.role is only user | model; OpenAI allows consecutive same-role messages and multiple system messages. Hoisting every system into one systemInstruction and mapping each message 1:1 to contents[] will 400 or scramble history on real Chat Completions payloads. Specify merge/split (and that thought text must not land in message.content).
| - OpenAI `tool_calls`/`tool`-role messages become Gemini `functionCall`/ | ||
| `functionResponse` parts; OpenAI `tools[]` (JSON-schema function defs) | ||
| become `tools: [{functionDeclarations: [...]}]`. |
There was a problem hiding this comment.
This is not a legal Chat Completions tool round-trip.
Gemini functionCall has name + args (object) and no id. OpenAI needs tool_calls[].id, string function.arguments, and role: tool messages with tool_call_id. Gemini functionResponse.response is a Struct, not a string.
The 2xx table below also maps finishReason: STOP → finish_reason: stop even when functionCall parts are present. OpenAI requires finish_reason: tool_calls in that case (this is already how apis/src/openai/translation and anthropic/to_openai treat the tool path).
Document: id synthesis, JSON stringify/parse both ways, and the tool_calls finish reason.
| - **2xx**: `response::from_generate_content_body` maps | ||
| `candidates[0].content.parts[]` (`text` / `functionCall`) to OpenAI | ||
| `choices[0].message.content` / `tool_calls[]`; `candidates[0].finishReason` | ||
| (`STOP`, `MAX_TOKENS`, `SAFETY`, `RECITATION`, `OTHER`) to OpenAI | ||
| `finish_reason` (`stop`, `length`, `content_filter`, `content_filter`, | ||
| `stop`); `usageMetadata.{promptTokenCount,candidatesTokenCount,totalTokenCount}` | ||
| to OpenAI `usage.{prompt,completion,total}_tokens`. |
There was a problem hiding this comment.
Gemini safety blocks are often HTTP 200, not 4xx.
promptFeedback.blockReason with empty/missing candidates is a successful generateContent response (and, on the stream, only the first SSE chunk). Mapping only candidates[0] — and treating missing candidates as a malformed frame (~L351) — turns those into errors or empty stops. OpenAI's contract for this is finish_reason: content_filter (or an error object), not a parse failure.
The FinishReason table is also a stale subset. Current Vertex includes BLOCKLIST, PROHIBITED_CONTENT, SPII, MALFORMED_FUNCTION_CALL, UNEXPECTED_TOOL_CALL, MODEL_ARMOR, IMAGE_*. MALFORMED_FUNCTION_CALL is a 200 with empty parts. Map them or give an explicit fallback; do not leave unmapped 200s looking like stop.
| - **non-2xx**: `error::to_openai_error` maps Gemini's error envelope | ||
| (`{"error": {"code", "message", "status"}}` — already wrapped in an | ||
| `"error"` object, unlike Bedrock's flat `{"message": ...}`) into OpenAI's | ||
| `{"error": {"message", "type", "code"}}`: `message` copies directly, | ||
| `status` (an enum string like `INVALID_ARGUMENT`) maps to `type`, `code` | ||
| (Google's numeric gRPC-style code) maps to OpenAI's `code` field. |
There was a problem hiding this comment.
error.code in Google REST JSON is not a gRPC numeric code.
AIP-193: error.code is the HTTP status (400, 401, 403, 429, 500); error.status is the google.rpc.Code name (INVALID_ARGUMENT). OpenAI's envelope is error.type = invalid_request_error | authentication_error | rate_limit_error | …, and error.code is a string or null.
Copying status → type and code → code produces type: INVALID_ARGUMENT / code: 400, which is not OpenAI. Map HTTP status → OpenAI type, same idea as error_type_for_status in apis/src/anthropic/to_openai/response.rs.
| Vertex-specific work is only what comes *after* frame reassembly: each | ||
| complete `SseFrame`'s `data` is Gemini's own streaming delta JSON — one | ||
| `candidates[0].content.parts[]` fragment plus, on the terminal frame, | ||
| `finishReason` and `usageMetadata` — translated into an OpenAI | ||
| `chat.completion.chunk` SSE frame, using the same field mapping as the | ||
| non-streaming response path (`response::from_generate_content_body`, | ||
| applied incrementally instead of once), followed by a final | ||
| `data: [DONE]\n\n`. A structurally malformed frame (not valid JSON, or | ||
| missing `candidates`) does not abort the whole response — it emits one | ||
| terminal OpenAI-shaped error chunk and stops, the same fail-safe-degrade | ||
| behavior `bedrock_stream_events` uses for a corrupted binary frame, | ||
| adapted from the same overflow/error-recovery philosophy behind the | ||
| token-accounting fix (praxis-proxy/ai#674). |
There was a problem hiding this comment.
Transport reuse (SseFrameParser + ?alt=sse) is fine. The payload mapping is not.
Each Gemini SSE event is a GenerateContentResponse: incremental parts[].text, finishReason empty until the last chunk, no [DONE]. OpenAI streaming is object: chat.completion.chunk with choices[].delta (role on the first chunk, content/tool-call argument fragments after that, delta: {} + finish_reason on the last), then data: [DONE].
from_generate_content_body applied incrementally emits non-streaming choices[].message objects as SSE data. OpenAI clients will not parse that. Specify a delta state machine; do not reuse the buffered mapper.
Also: missing candidates is not always a malformed frame — see promptFeedback on the 2xx path above.
|
We're closing this pull request because we've moved all proposals to praxis-proxy/enhancements. Please see our announcement for more details. We apologize for any inconvenience. Please retarget this PR and the new repository and let us know if you need any help. 🖖 |
6 participants
Summary
Adds the How? section for #779, one of the four per-provider sub-tasks split out of #762. Builds on the accepted What?/Why? in
docs/proposals/00762_api-translation.md, including its resolution that Vertex shares the SSE decoder family via a forced?alt=sse.vertex_translate,vertex_stream_events) mirroring the Bedrock design in #783.project/locationare operator-config-only and never read from the request at all (stronger than "strip");modelis independently allowlisted before path construction, same pattern as Bedrock.apis/src/openai/sse::SseFrameParser+HttpFilterContext::insert_filter_statepattern (already used byapis/src/openai/responses/stream_events/) for the byte-level SSE reassembly, instead of a new parser.credential_injectbearer-token seam for the upstream OAuth2 access token, with the shorter (~1h) GCP token lifetime explicitly flagged as a graduation criterion.Test plan