fix(llm/messages): normalize Claude Code in-conversation system messages#2015
Open
yalindogusahin wants to merge 1 commit into
Open
fix(llm/messages): normalize Claude Code in-conversation system messages#2015yalindogusahin wants to merge 1 commit into
yalindogusahin wants to merge 1 commit into
Conversation
…ges into top-level system Signed-off-by: Yalın Şahin <yalinsahin1@gmail.com>
2a8c517 to
53e2538
Compare
|
Yes actually I've tried two things before I decided to go with the merging into system (sacrificing the positional part).
Both ways completely messed up the models behavior (I was testing with Qwen series) so I decided to go with the the merging. I've inspected Claude's midway system messages - they're mainly to inject skills information, which is completely fine to be in system for now. |
Author
|
Thanks @Syraxius. Anything you'd change in this PR? Naming, normalization placement? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Claude Code 2.1.157+ injects in-conversation system reminders inside the Anthropic
messagesarray (messages[*].role == "system"), which is not part of the Anthropic Messages API spec —systemis a top-level field, andmessages[].roleis limited touser/assistant. Strict spec implementations like vLLM's Anthropic endpoint reject the payload before request conversion (see vllm-project/vllm#44048), and the same payload also fails agentgateway's typedRoleenum during messages → completions / bedrock translation.This PR normalizes the request at the proxy entry point so every downstream conversion path sees a canonical Anthropic shape:
messages[*].role == "system"entries frommessagessystemfield, preserving any existing top-level system content firstChanges
crates/agentgateway/src/llm/types/messages.rs: addRequest::normalize_system_messages()method handling both string and arrayContentBlockforms; merges extracted content into aTextBlock::Arrayafter existing top-level system. 6 new unit tests cover extraction, preservation order, array content blocks, no-op paths, and a regression test that exercises the actualcompletions::from_messages::translatecall with a Claude Code 2.1.157+ shaped payload.crates/agentgateway/src/llm/mod.rs: callnormalize_system_messages()insideLLM::process_messages_requestafter the request body is parsed and beforeprocess_requestdispatch. Single chokepoint covers OpenAI, Bedrock, Vertex, and Anthropic-native passthrough.crates/agentgateway/src/llm/conversion/completions.rs,crates/agentgateway/src/llm/conversion/bedrock.rs, andRequestType::to_anthropicin messages.rs: defensivenormalize_system_messages()calls in the direct library entry points so callers that bypass the proxy entry point (tests, embedders) get the same compatibility.Trade-off (out of scope here)
The merge loses positional information: every in-conversation system block is appended after existing top-level system content. Preserving position would require provider-aware translation — OpenAI chat completions natively supports multiple
systemmessages mid-conversation, Bedrock could usesystemblocks, Anthropic native cannot. A future opt-in flag (e.g.preserve_in_conversation_system) could expose this for backends that benefit from it. For now, normalizing into the standard Anthropicsystemfield matches the same approach vLLM took in vllm-project/vllm#44048 and gives the least surprising baseline behavior.Test Plan
cargo build -p agentgateway(passes locally)cargo test -p agentgateway llm::types::messages::normalize_system_messages_tests— 6/6 passed locally