feat: move eval execution from GitHub Actions to Vercel Sandboxes - #192
Merged
mattrossman merged 42 commits intoAug 14, 2026
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
mattrossman
changed the title
p-limit / p-retry
mattrossman
changed the title
p-limit / p-retryp-limit / p-retry
mattrossman
changed the title
p-limit / p-retryp-limit / p-retry
…he session ceiling
…sandbox-runner-for-evals' into mattrossman/ai-912-spike-vercel-sandbox-runner-for-evals
requireVercelCredentials and vercelCredentialsFromEnv checked the same 3 env vars separately, and the latter was re-derived from process.env on every concurrent pair. Compute it once in runPairs and thread it through instead.
4 was overly conservative for the actual benchmark suite (238 pairs today, growing). 250 is validated against a real full-benchmark run (#192) with clean completion and no permanent failures, with the 429 rate-limit retries handling the initial creation burst.
mattrossman
marked this pull request as ready for review
August 14, 2026 01:39
Rodriguespn
reviewed
Aug 14, 2026
Rodriguespn
reviewed
Aug 14, 2026
Rodriguespn
reviewed
Aug 14, 2026
Rodriguespn
reviewed
Aug 14, 2026
Contributor
|
nit: We should clean the |
isRetryableSandboxCreateError treated every non-APIError as retryable, so deterministic failures like a bad OIDC token got the full 12-attempt backoff budget on top of the pair-level retry. Now it bails on known credential errors, retries known network faults, and caps anything unrecognized to a couple of attempts with a warning so gaps in the categorization show up in logs.
… errors createSandbox now rethrows a definitive 4xx or credential error as p-retry's AbortError, which propagates through any pRetry call it bubbles through. That keeps runPairs' pair-level retry from wasting another attempt on a sandbox create that already failed for good.
vercelCredentialsFromEnv() now always requires explicit VERCEL_TOKEN, VERCEL_TEAM_ID, and VERCEL_PROJECT_ID instead of deferring to the SDK's OIDC resolution when VERCEL_OIDC_TOKEN happened to be set. That token only appears locally as a stale leftover from vercel env pull on an unrelated project (never used in CI, which has no id-token permission), and pnpm eval:vercel doesn't even load .env.local where it would live. Since Sandbox.create() now always gets full explicit credentials, the SDK's local credential-resolution errors (bad/missing OIDC token) can no longer be thrown, so the retry classifier's special-casing for them is removed too.
Adds timeout-minutes: 60 to run-evals (was defaulting to GitHub's 360-min ceiling), based on a real 237-pair run finishing in ~28 min under heavy retry contention. publish-results now runs even if run-evals times out or partially fails (needs.run-evals.result != 'skipped', with an explicit always() to bypass the implicit success() gate), so completed pairs still get published instead of stranded in a 3-day-retention artifact. A new "Audit pair results" step diffs expected pairs against what actually downloaded, posts a warning annotation and job summary on any gap, skips the unattended scheduled auto-merge when partial, and a final "Require complete results" step fails the job after publishing so an incomplete refresh can't go unnoticed. Also drops submodules: recursive from the run-evals/publish-results checkouts: carryover from the old per-pair GH-hosted matrix jobs, unneeded now that sandboxes clone and init their own submodules.
mattrossman
force-pushed
the
mattrossman/ai-912-spike-vercel-sandbox-runner-for-evals
branch
from
August 14, 2026 15:16
4bf8a84 to
117ec3a
Compare
Forces investigate-security-001-public-table to sleep 10 min so a manual dispatch can produce a real mixed success/failure outcome to verify the new eval-refresh.yml partial-results handling. Reverting before this PR is done.
mattrossman
force-pushed
the
mattrossman/ai-912-spike-vercel-sandbox-runner-for-evals
branch
from
August 14, 2026 15:18
117ec3a to
489b380
Compare
If every pair fails, run-evals never uploads a raw-results artifact at all (nothing to upload), so Download raw results hard-errors and skips every step after it, including the new Audit/Require steps meant to report exactly this. continue-on-error lets those steps still run and report clearly on a total failure instead of masking it as an unrelated artifact-download error.
Live-tested the total-failure path: with 0/N pairs succeeding, export-results legitimately errors on "no result files matched" and nothing gets committed, but the warning/summary/error text all unconditionally said "results were published" or "partial results published" regardless. Adds a genuinely-all-missing branch to Audit pair results and drops the false claim from Require complete results.
The eval-refresh.yml partial-results handling has been verified live via manual dispatches. Removes the throwaway test-only delay.
Collaborator
Author
@Rodriguespn I've removed the 2 sandbox related labels, along with the stale AI gateway opt-in label |
Rodriguespn
approved these changes
Aug 14, 2026
Rodriguespn
left a comment
Rodriguespn
left a comment
Contributor
There was a problem hiding this comment.
LGTM, thanks for addressing the feedback 🙌 Great work adding a partial results recovery mechanism!
claude Bot
pushed a commit
that referenced
this pull request
Aug 14, 2026
Brings in a week of upstream work, most notably #192 (eval execution moved from GitHub Actions to Vercel Sandboxes), #178 (claude-code MCP tool-name normalization), #168 (RLS docs guide eval) and several results refreshes. One conflict, in apps/framework/package.json: both sides extended the `check` script. Upstream added `test:vercel-runner` for the new Vercel runner tests; this branch added `test` to give the prompt-assembly tests a runner. Both are kept: check: pnpm typecheck && pnpm test && pnpm test:framework && pnpm test:vercel-runner The dependency block in that file merged cleanly (upstream re-sorted it and added @vercel/sandbox, p-limit, p-retry and zod). run-eval.ts, packages/core/src/agents/engine.ts and packages/core/src/index.ts merged without conflict and both intents are intact: - run-eval.ts now takes readFlag/positiveInteger from lib/cli-args.js (upstream hoisted the local readFlag out) while still assembling its prompt through buildSystemPrompt() and recording systemPrompt in the result JSON. - engine.ts still stages no system-prompt file when the assembled prompt is empty, and additionally passes upstream's mcpServerNames to parseTranscript. - index.ts keeps upstream's parser/transcript type changes alongside this branch's AgentRunArgs.systemPrompt and LocalStackSessionArgs.agent docs. No new CLI harness landed upstream (still claude-code, codex, opencode), so the `skills add --agent` list is unchanged. The new Vercel runner shells out to `pnpm eval`, so it inherits this branch's prompt behaviour with no change. apps/web/src/data/*eval-results.json and results/ are untouched by this branch and the merge takes main's versions wholesale. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves eval execution from GitHub Actions to Vercel Sandboxes. The GitHub workflow remains for orchestrating sandboxes via a new
run-vercel-evals.tsentrypoint. It uses p-limit and p-retry for concurrency limits and retry policies. Also considered effect, but our use case isn't very complex to justify the runtime.Sandbox limits
As of Aug 5th, our Pro plan supports 10,000 concurrent sandboxes and 5,000 vCPU allocated / min with dynamic allocation quotas that ramp up during sustained usage.
I'm starting with a default of 250 concurrent sandboxes for the workflow, chosen as a round number similar to our existing 256 matrix job limit, but we can very likely increase this as our benchmark + attempt counts grow.
Note on Sandbox retries
Sandboxes creation is subject to dynamic quotas on vCPU allocation limits. The Sandbox SDK already implements a basic retry policy, but it's not the most forgiving, so we add our own exponential backoff retries on top.
Vercel project/creds
Repo is currently wired up to the evals-runner project on Vercel via
VERCEL_PROJECT_ID,VERCEL_TOKEN,VERCEL_TEAM_ID, a carryover from the Vercel Workflows spike. This project is configured w/ the 3 provider API keys.If desired, we could move sandbox usage back to the evals project since we're not deploying any new application entrypoints with this PR.
Job artifacts
Previously each matrix job in
run-evalsuploaded a separate results artifact which we'd then download and combine. Now, since there's only onerun-evalsjob we upload a single combinedraw-resultsartifact.Job logs
I intentionally don't wait for
command.logs()because it sometimes loses connection and drops output. Instead, I consumecommand.output('both')so you'll see theRUNandPASS/FAIL'appear only at the end of each sandbox's eval execution, which seems like a reasonable tradeoff.Sample runs:
Results look normal after benchmark refresh (Preview):
How to test
I test by manually dispatching the refresh workflow against this branch, though it's a slow / compute heavy thing to refresh the full suite, so you can probably just review the existing runs or dispatch a smaller refresh.
Next steps
Currently, benchmark execution takes ~33 min, comparable to our latest refresh on the GitHub Actions matrix. From here, we're planning to parallelize attempts instead of giving agents 2 attempts in sequence. Depending on how many sandboxes this ends up spawning, it'll potentially decrease execution time as attempts can run in parallel.
Another optimization I haven't landed here is the warm boot trick from Pedro's #114, I'll likely revisit that as a follow up since it hasn't regressed overall benchmark duration yet.
Related:
Closes AI-912