test(evals): 2026-08-08 arena re-run — mention-gated kickoff option, delegate-and-forward repinned to the shipped contract, baseline refreshed - #790
Open
Poytr1 wants to merge 3 commits into
Conversation
…mes (local measurement run)
…he arena
A user asked agent A to "send hello to agent b and forward reply". A woke B with
`needsReply`, polled the returned child in the same turn, read `in-progress`, and
then told the user "Agent B completed its turn but returned no message to
forward" — a completion claim its own last observation contradicted.
`needsReply` is a two-sided contract with only one side stated: the CHILD gets a
standing report-back directive, the PARENT gets `{ok, wake, childSessionId}` and
no statement that the call is asynchronous. `viewSessionStatus` then advises
"prefer waiting for the child's reply", which no turn can do.
Two layers:
- `evals/test/delegate-and-forward.test.ts`, credential-free and added to
`pnpm eval:collab:contracts`. Two green characterization pins record the surface
as it is (the wake result's exact key set; a same-turn poll returning
in-progress against a provably mid-turn child, via an explicit rendezvous). Five
`it.fails(…)` pin the affordances that must change, each naming the file: the
async contract in the wake result, the impossible "prefer waiting" advice, the
missing "when polling IS appropriate", `done` conflating "turn ended" with
"reported back", and a headless child's answer being dropped silently.
- `evals/test/delegate-and-forward-real.test.ts`, real ACP runtime, NOT in the
gate, reported as a rate over trials with transcripts under `.artifacts/`.
Measured over 5 trials of real local Claude Code (sonnet): the parent-side failure
did NOT reproduce — 0 status polls, 0 premature claims, A ended its turn saying it
would forward. A different failure did, 2 of 5: a postless child session is
headless, so a child that answers in prose instead of calling
`sendMessage {sessionId}` has its answer discarded with no signal, and the parent
waits forever. From the parent's seat that is literally "returned no message to
forward".
`RoutingFixture` gains real-subject support (no hostFactory, template preflight,
per-seat description override), per-turn tool-call traces attributed by the
daemon's own turnId, and the daemon's peer-wake delivery records.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ntract; record the 2026-08-08 arena re-run The parent-side half of the needsReply async contract shipped on main: the wake result carries reply/nextAction/message, viewSessionStatus is diagnostic-only, and SessionStatusResult.reply.state separates "turn ended" from "reported back". The former red pins are now green guard tests so none of it can regress; the one genuinely open pin — a headless child's prose answer silently dropped — stays expected-fail, and the 2026-08-08 real-model trial reproduced it (1/1). The baseline doc gains §5.6 (full real-model re-run, one trial per case, mention-gated production kickoff for the counting games), corrects §6.1 (the 16-edge automatic-turn bound is a scripted-speed artifact; a real 2-agent chain is bound by the hop cap — measured 20/20 with 19 hops used, 0 gated), and fixes the real-subject recipe (npx-launched user runtimes are filtered as not-installed; the runtime id must be claude-acp for the verified memory off-switch; launch the adapter via node). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wrap-up of the 2026-08-08 Collaboration Arena re-run on current
main: test/docs housekeeping only — no product code changes.What's in here
Mention-gated production kickoff for the counting games (
evals/games/): an opt-inbindMatch: 'mention'for the peer/quota counting games — the production shared-channel convention the routing-acceptance fixture always used — with the kickoff entering as an ordinary human platform message that @mentions every participant. Defaultautounchanged; all existing contract tests unaffected.PR test(evals): encode the delegate-and-forward async-contract case in the arena #737's delegate-and-forward case, rebased onto current
mainand repinned to today's truth (cherry-picked with original authorship; supersedes test(evals): encode the delegate-and-forward async-contract case in the arena #737 — recommend closing it):needsReplyasync contract shipped on main since test(evals): encode the delegate-and-forward async-contract case in the arena #737 was written: the wake result carriesreply/nextAction: 'finish-turn-and-wait'/ contract prose,viewSessionStatusis described as diagnostic-only and no longer advises "waiting", andSessionStatusResult.reply.statedistinguishes "the child's turn ended" from "the child reported back". The formerit.failspins for all of that are now green guard tests (5 of them) so none of it can regress. No change to the shipped wake result — it is pinned exactly as shipped.{ok, wake, childSessionId}result is updated to pin the shipped surface (exact key set + same-turn poll truthfulness).nextActionframing answers that question differently, and demanding the old phrasing no longer describes a deficiency.it.fails): a headless child that answers in prose instead ofsendMessage {sessionId}still has its answer silently dropped. Verified still-red againstmain, and the 2026-08-08 real-model trial reproduced it (1/1): parent behaved perfectly under the new contract (0 polls, no premature claim) and the reply was still lost on the child side.delegate-and-forward-real.test.ts, env-gated, not in CI) is unchanged.Baseline doc updated (
docs/designs/collaboration-arena-baseline.md):npx-launched user runtimes are now filtered as not-installed (PACKAGE_LAUNCHERS), and the runtime id must beclaude-acpto match the verified memory off-switch policy — install the adapter and launch it vianode. A working template shape is included.Verification
pnpm typecheck— clean (afterpnpm build)pnpm lint— cleanpnpm eval:collab:contracts— 16 files, 120 passed + 1 expected-fail (121)pnpm eval:contracts— 46/46pnpm --filter @agentconnect.md/daemon test— 2880 passed; 2 failures are pre-existing and unrelated (acp-matrixlive-runtime matrix needs local real runtimes;daemon-agent-mention-routinghop-cap case is a load-sensitive 5 s timeout) —git diff origin/main -- packages/is empty, so no daemon source changed here.🤖 Generated with Claude Code