Skip to content

fix(acp): deliver plain-text replies for self-hosted OpenAI-compatible agents - #4874

Open
repudi8or wants to merge 6 commits into
block:mainfrom
repudi8or:fix/deliver-plain-text-replies
Open

fix(acp): deliver plain-text replies for self-hosted OpenAI-compatible agents#4874
repudi8or wants to merge 6 commits into
block:mainfrom
repudi8or:fix/deliver-plain-text-replies

Conversation

@repudi8or

@repudi8or repudi8or commented Aug 5, 2026

Copy link
Copy Markdown

Summary

  • Small local models served over a self-hosted OpenAI-compatible endpoint (e.g. llama.cpp/vLLM/mesh-llm) routinely finish a multi-step turn by writing the answer as prose instead of calling send_message — the turn reports EndTurn successfully, the ACP activity log shows the correct reply, but nothing is ever published to the channel. This is what we observed running a self-hosted local-model agent against a mesh-llm deployment.
  • crates/buzz-acp: retain each turn's streamed text (bounded at 8 KiB) plus whether a publish tool call was seen; on EndTurn, if nothing was published, deliver that text as a threaded channel reply instead. Gated strictly on EndTurn (not MaxTokens/MaxTurnRequests, which mean the turn was truncated) and filters bare acknowledgements ("OK", "Done"). Off by default via --deliver-plain-replies / BUZZ_ACP_DELIVER_PLAIN_REPLIES, since cloud models reliably call send_message themselves and enabling it there would risk double-posting.
  • desktop/src-tauri: opt the built-in "Buzz shared compute" (relay-mesh) preset into this fallback, and separately default it on for the general openai-compat provider too (a self-hosted endpoint configured directly, not through the built-in preset) — scoped narrowly so plain openai (the real OpenAI cloud API) is untouched. An explicit user override (including an intentional false) always survives.
  • crates/buzz-agent: while validating the openai-compat fallback above against a real self-hosted setup (a proxy fanning a single model alias out across several backend providers), we also hit a second, unrelated failure mode worth fixing alongside it — a 400 from a backend routed through OpenRouter: "auto tool schema uses unsupported assertions or reserved metadata". The root cause: buzz-dev-mcp's tool schemas are auto-derived via rmcp/schemars, which routinely emits JSON Schema metadata and assertion keywords ($schema, title, format, additionalProperties, range/length constraints, ...) that Anthropic and the official OpenAI API happily ignore, but that some openai-compat backends validate strictly and reject outright — breaking every tool call for that agent, not just the reply-delivery path. Added sanitize_tool_schema, applied once at tool registration alongside the existing schema-size cap (cap_schema), which strips a denylist of those keywords from the whole schema tree before it's ever sent to a provider. Every stripped key only narrows or documents what a schema accepts, so this can only loosen a backend's validation — it can't change what a tool call actually does, and structural/semantic keywords (type, properties, required, items, description, enum) are left untouched. This is the same underlying theme as the reply-delivery fix above (self-hosted openai-compat backends don't share the reference providers' tolerances), so it seemed worth landing together — useful for anyone pointing buzz-agent at vLLM, llama.cpp, a litellm/multi-provider proxy, or any OpenRouter-routed model with a stricter function-calling schema validator than Claude/GPT.
  • desktop/src-tauri: merged main in to pick up two commits that had landed on crates/buzz-acp since this branch started, and split readiness.rs's test module (readiness_tests.rs / readiness_provider_tests.rs) to stay under the desktop file-size ratchet after that merge — no behavior change, same tests, same assertions.

Test plan

  • cargo test --manifest-path desktop/src-tauri/Cargo.toml --lib — 2250 passed, 0 failed, 14 ignored (full crate, not scoped to touched files), run independently by two people
  • New targeted tests: publish_detection_matches_tool_and_shell_shapes, bare_acknowledgements_are_not_worth_publishing, non_mesh_provider_does_not_opt_into_plain_reply_delivery, openai_compat_provider_opts_into_plain_reply_delivery, real_openai_provider_does_not_opt_into_plain_reply_delivery, explicit_deliver_plain_replies_override_is_preserved_for_openai_compat
  • cargo test -p buzz-agent --lib — 401 passed, 0 failed, including 4 new schema-sanitization tests (sanitize_tool_schema_strips_root_metadata, sanitize_tool_schema_strips_nested_property_assertions, sanitize_tool_schema_preserves_structural_and_semantic_keywords, sanitize_tool_schema_recurses_into_arrays)
  • cargo clippy -p buzz-agent -p buzz-acp --lib -- -D warnings and cargo fmt --manifest-path desktop/src-tauri/Cargo.toml --all -- --check clean
  • Full pre-push suite green after the main merge: branch-skew, rust-tests, desktop-check (file-size ratchet), desktop-typecheck, desktop-test (4480 JS tests), mobile-test, desktop-tauri-checks (2250 Rust tests)
  • Independent code review of the diff
  • Live end-to-end repro against the original self-hosted agent — not done pre-merge: the fix spans buzz-acp (bundled inside the Buzz Desktop app) and desktop/src-tauri, and there's no way to swap just one agent's binary without patching the signed Desktop app bundle used by every other agent on that machine. Recommend validating live once this ships through the normal Desktop build/update path.

Diagnosed and coordinated by the buzz-acp team (architect, dev, tester, conductor) in the team's working channel; core mechanism forward-ported from an unmerged branch (micspiral/mesh-0-74-gemma, commit bb4af4189) by Michael Neale.

with a helping hand from Claude Code

michaelneale and others added 2 commits August 5, 2026 16:27
buzz-agent's contract is that output is tool calls: streamed assistant text
is observability only and never published. Capable models honour that and
call `send_message`. Small local models served over shared compute do not
always — on multi-step turns they run their tools, then write the answer as
prose and end the turn, so the turn reports success with nothing in the
channel.

Retain the turn's streamed text in AcpClient (bounded at 8 KiB) alongside a
flag for whether a publish tool call was seen, and on EndTurn publish that
text as a threaded channel reply when the agent published nothing itself.
Gated on EndTurn only: MaxTokens / MaxTurnRequests mean the turn was
truncated, so the text is not a deliberate answer. Bare acknowledgements
("OK", "Done") are filtered, and taking the buffer clears it so the same
text can never post twice.

Off by default via --deliver-plain-replies / BUZZ_ACP_DELIVER_PLAIN_REPLIES:
for cloud models, which reliably publish their own replies, posting streamed
text would double-post. The desktop shared-compute preset opts in, so this
stays scoped to the case that needs it.

Verified end to end against a live relay with stub ACP agents that pin each
branch deterministically: a prose-only agent gets its answer delivered, and
an agent that calls send_message while also streaming prose posts exactly
once with the fallback silent.

Forward-ported onto main from the unmerged origin/micspiral/mesh-0-74-gemma
branch (bb4af41). This exact bug was hit in production by the "Local LLM
(llm1)" self-hosted mesh-llm/llama-server agent: turns ended with EndTurn
and correct prose visible in the ACP activity log, but nothing was published
to the channel because send_message was never called.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
with a helping hand from Claude Code
Signed-off-by: Brett Meehan <repudi8or@gmail.com>
…ly delivery

The plain-reply-delivery fallback landed for the built-in relay-mesh
("Buzz shared compute") preset only, gated on provider == "relay-mesh"
in apply_relay_mesh_env. A self-hosted model reached the same way small
local models always are — the "OpenAI-compatible" provider pointed at a
user's own llama.cpp/vLLM/mesh-llm endpoint instead of the built-in
preset — never opted in, so the exact same silent-drop bug reproduced:
the model answers in prose, ACP's activity log shows a correct reply,
and nothing is ever published to the channel.

Default BUZZ_ACP_DELIVER_PLAIN_REPLIES to "true" for effective provider
"openai-compat" too, in the same effective-env assembly step, right
after the relay-mesh translation. Scoped narrowly to "openai-compat":
plain "openai" (the real OpenAI cloud API) is left untouched, since
cloud models reliably call send_message themselves and enabling the
fallback there would risk double-posting. An explicit user-set value
(including an intentional "false") is preserved, matching the emptyness
convention used elsewhere in this module.

Reproduced against the "Local LLM (llm1)" agent in production: provider
"openai-compat", OPENAI_COMPAT_BASE_URL pointed at a remote llama-server
instance — confirmed via the agent's own config screenshots that it is
not on the relay-mesh preset, so relay_mesh.rs's existing default never
applied to it.

with a helping hand from Claude Code

Signed-off-by: Brett Meehan <repudi8or@gmail.com>
@repudi8or
repudi8or requested a review from a team as a code owner August 5, 2026 10:37
@repudi8or

repudi8or commented Aug 5, 2026

Copy link
Copy Markdown
Author

@wesbillman this fixes a bug affecting self-hosted OpenAI-compatible agents (e.g. local llama.cpp/mesh-llm endpoints) silently dropping their replies instead of publishing them — full context and test results in the PR description. Would appreciate a review/merge when you have a moment. I only have read access to block/buzz directly (opened from a fork), so can't self-merge.

@repudi8or

Copy link
Copy Markdown
Author

Live end-to-end repro: confirmed ✅

Independently verified the events directly on the relay (not just trusting the report):

  • Hello: 92df0f20db6cea23f2744e6b24dc476e09aae3a66299168e302fe2edf97abd0a@Local LLM (llm1) hello, tagged to the dev-branch test agent's pubkey 5299bb38ce49b5090c08022359842c396398a6b36efd4cbcd8484d1aa56b22eb.
  • Reply: 2351d7ad8b56f94ac457be5e880f5714dd3c59a45bf201a5f9343091f2339de8 — kind 9, from 5299bb38…, with an e-tag reply reference back to the hello event. Correctly threaded.

This closes the one unchecked item in the test plan: a self-hosted openai-compat agent's plain-text turn (no send_message tool call — the model never called it) landed as a real threaded channel reply instead of vanishing after generation. Setup: repudi8or's just dev isolated Desktop build (already on fix/deliver-plain-text-replies) spun up its own test copy of the "Local LLM (llm1)" agent against the real production relay, so this exercised the actual shipped code path end-to-end — not a mock.

Non-blocking follow-up spotted in the reply content (not the delivery mechanism, which is what this PR fixes): the model looped through many near-duplicate self-summaries for the full turn, and the delivered text is hard-cut mid-word — consistent with hitting the 8KiB MAX_TURN_TEXT_BYTES per-turn buffer cap in acp.rs. Delivery/threading itself is correct; this is llm1's own response quality (repetition / no natural stop) and worth a separate issue (better stop behavior or a repetition guard for small local models), not a blocker here.

rmcp/schemars-derived tool schemas (buzz-dev-mcp's shell, read_file,
str_replace, etc.) routinely carry JSON Schema metadata and assertions
like $schema, title, format, and additionalProperties. Anthropic and
the official OpenAI API tolerate these, but some openai-compat
backends reached via a self-hosted endpoint or a multi-provider proxy
(observed via a litellm-fronted OpenRouter route) hard-reject the
whole tool-call request with "unsupported assertions or reserved
metadata" instead of ignoring the extra keys.

Add sanitize_tool_schema, applied once at tool registration in
mcp.rs alongside the existing cap_schema size cap, to strip a
denylist of pure-metadata and format/range-assertion keywords from
every object in the schema tree. Every stripped keyword only narrows
or documents what the schema accepts, never widens it, so removal
can loosen validation but never break a previously-valid tool call.
Structural/semantic keywords (type, properties, required, items,
description, enum) are left untouched.

Signed-off-by: repudi8or <repudi8or@gmail.com>
…t-replies

Signed-off-by: repudi8or <repudi8or@gmail.com>
An earlier commit on this branch grew readiness.rs past its grandfathered
1742-line ratchet allowance (to 1850 lines), blocking the pre-push
desktop-check hook. Following this repo's existing convention for
oversized test modules (migration.rs's migration_tests.rs,
migration_command_tests.rs, etc., and readiness.rs's own
readiness_goose_file_config_tests.rs), extract the inline `mod tests`
block into sibling files declared via #[path]. The extracted module
still exceeded the 1000-line cap for new files, so it's split further
into readiness_tests.rs and readiness_provider_tests.rs. readiness.rs
itself drops to 690 lines.

Also includes a cargo fmt fix to relay_mesh.rs (a pre-existing line-wrap
picked up by running fmt across the whole crate), and reformats the two
new test files to match.

Signed-off-by: repudi8or <repudi8or@gmail.com>
Resolves conflicts in buzz-acp's turn/usage tracking fields and the
desktop relay-mesh env/readiness modules, restoring the branch's
plain-text-reply delivery fallback (relay-mesh auto/model wiring,
readiness test split) on top of main's newer usage-tracking and
relay-mesh model-translation changes.

with a helping hand from Claude Code

Signed-off-by: Brett Meehan <repudi8or@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants