Ignore flaky tools-call-sampling in conformance CI

[jhrozek]

Summary

Why: the tools-call-sampling MCP conformance scenario flakes ~1/3 of runs and intermittently reds every PR's E2E Tests / MCP Conformance check. Root cause is an upstream reference-server bug, not a ToolHive defect (filed as modelcontextprotocol/conformance#407): the everything-server sends its server→client sampling/createMessage on the client's standalone GET /mcp SSE stream with no relatedRequestId, so it races the tools/call handler; when the handler wins, the SDK drops the request on the not-yet-connected standalone stream (stores-and-silently-returns) and the client times out at 60s. Direct-to-server it's ~0% flaky; a proxy in front (ToolHive) widens the race.

What: post-filter the suite result in run-conformance.sh to ignore only tools-call-sampling (whether it passes or fails), while every other scenario still gates the job. expected-failures.yaml can't express a flaky scenario — listing it there stale-fails the ~2/3 of runs where it passes — so the exclusion is done in the harness with a clear revert marker.

Type of change

• CI / test infrastructure

Test plan

• Shell syntax check (bash -n) passes.
• Self-tested the parse: a suite whose only unexpected failure is tools-call-sampling → job passes; tools-call-sampling + any other unexpected failure → job still fails.

Does this introduce a user-facing change?

No.

Special notes for reviewers

Temporary. Remove this block once modelcontextprotocol/conformance#407 is fixed upstream and CONFORMANCE_VERSION is bumped to a release containing the fix (there is currently no such tag — the bug is present at v0.1.16 and at upstream HEAD). Related ToolHive work: #5886 / #5887 (ingress Squid standby=2, which reduces but cannot eliminate the client-side ordering race — the residual is upstream-only).

Generated with Claude Code

[JAORMX]

LGTM — a well-reasoned, correctly-scoped quarantine. ✅ I traced the harness logic and it's precise:

• PIPESTATUS[0] captures npx's real exit code (not tee's), read immediately after the pipeline — correct under pipefail.
• Only tools-call-sampling is ignored: others = the unexpected-failure set minus that one scenario (and blank lines), and it exit 0s only when unexpected is non-empty and others is empty. Any other unexpected failure → exit "${suite_rc}", so the job still gates on every other scenario.
• Fail-safe on non-scenario failures: if the suite fails without a parseable "Unexpected failures" list (crash/infra/timeout), unexpected is empty, the -n guard is false, and it falls through to exit "${suite_rc}" — the quarantine can't swallow a non-sampling failure.
• Harness-filter over expected-failures.yaml is the right call — a flaky entry there would stale-fail the ~2/3 of runs where the scenario passes; the pass-or-fail harness filter avoids that.

The root-cause writeup (upstream everything-server sending sampling/createMessage on the standalone GET SSE stream with no relatedRequestId, racing the handler → conformance#407, not a ToolHive defect) matches everything we saw across #5839/#5884/#5887, and cleanly closes the investigation: it's an upstream client-side ordering race, which a fronting proxy widens but doesn't cause.

One note (inherent to quarantining, acceptable given it's temporary): while this is active, a genuine ToolHive-side sampling regression wouldn't be caught by conformance CI either. The revert is gated on a manual CONFORMANCE_VERSION bump once #407 ships — the clear revert marker + issue links are good; a tracking issue to remove the block would make sure it doesn't outlive the upstream fix.

Approving — this unblocks everyone's MCP Conformance check without masking real failures.

_{🤖 AI-assisted review via Claude Code (CI/test + the flake context from #5839/#5884/#5887). Verified against 5b983ce.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.51%. Comparing base (db4ecb1) to head (5b983ce).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5888      +/-   ##
==========================================
+ Coverage   71.50%   71.51%   +0.01%     
==========================================
  Files         695      695              
  Lines       71005    71005              
==========================================
+ Hits        50774    50782       +8     
+ Misses      16573    16569       -4     
+ Partials     3658     3654       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.