08 — Cross-runner parity test suite

[gewenyu99]

Epic: #520 · Behavior change: none (CI) · Depends on: #524

Summary

A suite that runs the same framework apps under each runner (anthropic vs pi) and compares behavior.

Scope

• Drive each runner over the e2e test apps (wizard-workbench), selecting it via the existing
  WIZARD_CI_FLAG_OVERRIDES='{"wizard-runner":"<anthropic|pi>"}' on the build:ci channel (add
  "wizard-orchestrator":"true" to exercise the orchestrator router once it lands).
• Compare: run success, key tool effects (Edit/Write results), and that the blocked-action corpus stays
  blocked under each runner.
• Emit a per-runner scorecard for the 07 — Rollout + per-runner observability (canary via wizard-runner) #527/09 — Decide, ramp, and decouple #529 decision.

Acceptance criteria

• Suite runs both runners in CI.
• Diffs/scorecard emitted; security corpus asserted on each.

Files

• parity test harness (wizard-workbench CI + wizard)

[gewenyu99]

Parity test plan (switchboard reshape, #701)

Repeatable override → snapshot test using the #779 --harness/--sequence overrides as the driver. Target: e2e-tests/test-applications/nextjs-app-router-test-app, via the e2e-harness/ TUI-capture + wizard-ci-driver.

Scenarios

#	Invocation	Resolves to
1	wizard (no overrides)	linear / anthropic / sonnet
2	wizard --harness pi --sequence linear	linear / pi / sonnet
3	wizard --harness pi --model openai/gpt-5 --sequence linear	linear / pi / openai/gpt-5

Decision taken: scenario 3 needs a model selector — adding a --model override (third axis alongside --harness/--sequence, prod-gated + tree-shaken; cliModel middleware overlaying pick.model). Completes the epic's "per-program model selection" as a user-reachable config.

Orchestrator guard (no run): unit test pinning resolveSequence → linear with no --sequence/no flag (sequence-axis coverage, complements the harness precedence tests in switchboard.test.ts); + prod-build gating assertion that --sequence is rejected.

Deliverables: (1) durable snapshot regression tests in e2e-harness/ (scenarios 1–3); (2) live runs → an app PR per scenario with the wizard's changes + screenshot, linked in #520.

Merge-readiness audits: #788.