Storyboard runner: webhook_receiver_envelope grades "Failing" when receiver URL not configured; yaml states it should grade not_applicable

[kapoost]

Repro

• Agent registered on AAO panel: any AdCP agent (in my case signals.purrsonality.rocketscience.pl/mcp).

• Agent does NOT expose a webhook receiver — it is a signals agent, not a buyer/orchestrator.

• Panel's compliance run is not configured with webhook_replay_receiver.url.

• AAO panel shows the storyboard as:

  UNIVERSAL: WEBHOOK-RECEIVER-ENVELOPE — Webhook receiver envelope: inbound POST conformance · 5 steps · Failing · 0/5 steps

  First failed step: POST delivery report inside MCP webhook envelope
  post_delivery_report_inside_mcp_webhook_envelope · replay_webhook_vector
  No webhook replay receiver URL configured. Pass webhook_replay_receiver.url in StoryboardRunOptions to run inbound receiver conformance.
  Showing the first failing step; 4 more failures remain.

The storyboard's own narrative (compliance/cache/<v>/universal/webhook-receiver-envelope.yaml) is explicit:

The storyboard grades not_applicable when the runner was not configured with a buyer receiver URL. It is for buyer/orchestrator receiver testing, not for seller endpoint grading.

Why the step returns are actually correct

dist/lib/testing/storyboard/runner.js executeReplayWebhookVectorStep handles the missing URL correctly at the step level:

```js
if (!receiver?.url) {
const detail = 'No webhook replay receiver URL configured. Pass `webhook_replay_receiver.url` in StoryboardRunOptions to run inbound receiver conformance.';
return {
step_id: step.id,
// ...
passed: true,
skipped: true,
skip_reason: 'grader_skipped',
skip: buildSkip('not_applicable', detail),
// ...
};
}
```

Each step returns `passed: true, skipped: true, skip.reason: 'not_applicable'`. Correct per storyboard yaml.

Where the bug is

The storyboard-level status rollup (or AAO panel's aggregation over that rollup) is treating a storyboard whose 5/5 steps are `passed: true, skipped: true, reason: not_applicable` as Failing 0/5 rather than as Not applicable or Silent.

Peer storyboards using the same skip pattern (`applicability_gate_missing`, `required_any_of_tools` unmet) do surface with `overall_passed: true, failed_count: 0, skipped_count: 1` and don't render as "Failing" on the panel. This one does. That's the inconsistency.

Expected

Storyboard-level status: Not applicable (or Silent) with `failed_count: 0`. Panel headline should not read "Failing", and "5 failures" should not be counted.

Actual

Panel headline: "Failing 0/5 steps", counts 5 failures, tells the operator to "review POST delivery report inside MCP webhook envelope, then re-test." There is nothing on the operator's side to fix — the agent under test isn't a webhook receiver and the runner has no receiver URL to replay against, exactly the case the storyboard yaml declares `not_applicable`.

Impact

Any agent that is not a buyer/orchestrator webhook receiver will show a spurious "Failing" storyboard on its AAO card, degrading the badge card presentation for otherwise clean signals / seller / creative / governance agents. Not badge-blocking today, but flagged in "Needs attention" and "1 storyboard not passing" counters, which distracts from real issues.

Encountered on `signals.purrsonality.rocketscience.pl/mcp` (Signal-owned agent, 3.0 + 3.1 badged, 15-day streak) — "Needs attention: 1 storyboard not passing. Start with webhook_receiver_envelope" while no other core / signals / error_handling storyboard has any actual issue.

Fix direction

Either:

• A. Storyboard-level rollup: when 100% of steps in a storyboard are `skipped: true` with `skip.reason: not_applicable`, storyboard status classifies as `not_applicable`, not `failing`. AAO panel then renders it as such (or hides the card).
• B. Runner: when no `webhook_replay_receiver` is configured AND the storyboard's declared applicability requires one, skip the whole storyboard rather than executing each step to per-step `not_applicable`. Storyboard status `skipped` naturally.

Option B mirrors what other capability-gated storyboards already do (`required_any_of_tools`, `required_capabilities` gates in `compliance/comply.ts`), and would be the cleaner fix.

Session

Discovered while sweeping AAO panel for badge-blocking issues after the 11.2.0 upgrade sweep.

[bokelley]

Triage

Classification: Bug
Bucket(s): conformance
Status: ready-to-implement

What the experts said:

• ad-tech-protocol-expert: Valid bug — requiredPhaseHasExecutedPass is false because all steps are skipped, so overall_passed: false even though failed_count: 0; contradicts the storyboard yaml's not_applicable intent. Option B (pre-flight gate) is architecturally correct, mirrors the existing webhook_receiver pattern exactly.
• code-reviewer: Valid bug, low complexity, non-breaking. Option B is the right fix. Adds a unit test shape that follows the existing webhook_receiver gate fixture.

My take: Both experts agree. The webhook_replay_receiver option has no pre-flight gate in detectImplicitRequires, so the runner executes all 5 steps, each skipping individually, then the rollup's requiredPhasesPassed check (which requires !s.skipped && s.passed) computes false → overall_passed: false. All other applicability skips (required_any_of_tools, required_capabilities, missing_tool) short-circuit to a synthetic result before any steps run. This storyboard needs the same treatment.

Implementation scope:

• detectImplicitRequires (~line 1690, runner.ts): scan phases for step.task === 'replay_webhook_vector'; if found, push 'webhook_replay_receiver' to the implicit requires list (mirrors the valueContainsWebhookToken check for webhook_receiver).
• RequirementName union + KNOWN_REQUIREMENTS set (types.ts): add 'webhook_replay_receiver'.
• resolveRequirement switch (~line 1529, runner.ts): add case 'webhook_replay_receiver': — return unmet when !options.webhook_replay_receiver?.url.
• REQUIREMENT_TO_SKIP_REASON (~line 1329, runner.ts): add webhook_replay_receiver. Skip reason: 'not_applicable' (this is an agent-type mismatch — signals/seller agents are architecturally not receivers — not a misconfiguration; aligns with storyboard yaml intent). Note: 'requirement_unmet' is also defensible if the reader treats absent config as an operator omission; implementer should pick one and confirm with the storyboard yaml narrative.
• Test: add a unit test — minimal synthetic storyboard with a single replay_webhook_vector step, run without webhook_replay_receiver, assert overall_passed: true, skipped_count: 1, failed_count: 0, phases[0].steps[0].skip.reason === 'not_applicable'. Follow the existing webhook_receiver gate fixture as the template.
• Non-breaking: confirmed. Agents running with a valid webhook_replay_receiver.url hit none of the new gate code; only the no-URL path changes from failing 0/5 → not_applicable.
• Run npm run ci:quick before pushing. No schema or public API changes; no ci:schema-check / ci:docs-check needed. Changeset required (conformance runner behavioral fix → patch).

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01XJ4gPdTzWoVtgsTz1o7gR6

---

Generated by Claude Code