feat(framework): pnpm local compare against the published baseline - #195
Open
barryroodt wants to merge 1 commit into
Open
feat(framework): pnpm local compare against the published baseline#195barryroodt wants to merge 1 commit into
barryroodt wants to merge 1 commit into
Conversation
pnpm local compare <eval...> [--experiment <id>] [--runs N] [--mcp <path>]
pnpm local compare --suite <regression|benchmark> [same flags]
Runs the treatment arm exactly as `run` does, then diffs it against the newest
published result for that eval and experiment on origin/main, and exits 1 when
the treatment regressed from PASS to FAIL.
It says out loud what it is. A flip is a SCREEN, not causal proof: the published
arm ran in the scheduled CI world, with the published mcp package, the
production docs index, and whatever model state existed at refresh time. So the
output prints that caveat, and the receipt records the published result's
commit, its first parent, and its age in days. The confound is explicit in the
artifact rather than left for the reader to remember.
Refusals stay pre-spend. An eval with no published row for the requested
experiment is refused before any model call, listing the experiments that ARE
published for it and pointing at `run` when there is no baseline at all.
`--suite` expands to every eval the published export carries for the
experiment, and is rejected in run mode or alongside explicit ids.
`--runs` defaults to the published row's attempt count, so the arms match
unless you say otherwise.
Verified: smoke 19/19 (8 new), tsc and biome clean.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 3/5 for #128. Base: #194.
Runs the treatment arm exactly as
rundoes, then diffs it against the newest published result for that eval and experiment onorigin/main. Exits 1 when the treatment went PASS -> FAIL.It says out loud what it is
A flip is a screen, not causal proof. The published arm ran in the scheduled CI world: published mcp package, production docs index, whatever model state existed at refresh time. So the output prints that caveat, and the receipt records the published result's commit, its first parent, and its age in days. The confound lives in the artifact rather than in the reader's memory.
That is the one design decision here I would push back on changing. A tool that printed
PASS -> FAILwithout it invites exactly the wrong inference.Refusals stay pre-spend
An eval with no published row for the requested experiment is refused before any model call, listing the experiments that are published for it and pointing at
runwhen there is no baseline at all.--suiteexpands to every eval the published export carries for the experiment, and is rejected in run mode or alongside explicit ids.--runsdefaults to the published row's attempt count, so the arms match unless you say otherwise.Verified
tsc --noEmitandbiome checkclean