feat: evals runner with vercel workflows, sandbox - #186
Draft
mattrossman wants to merge 12 commits into
Draft
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
mattrossman
added a commit
that referenced
this pull request
Aug 14, 2026
Moves eval execution from GitHub Actions to [Vercel Sandboxes](https://vercel.com/docs/sandbox). The GitHub workflow remains for orchestrating sandboxes via a new `run-vercel-evals.ts` entrypoint. It uses [p-limit](https://www.npmjs.com/package/p-limit) and [p-retry](https://www.npmjs.com/package/p-retry) for concurrency limits and retry policies. Also considered [effect](https://effect.website/), but our use case isn't very complex to justify the runtime. **Sandbox limits** [As of Aug 5th](https://vercel.com/changelog/vercel-sandbox-now-supports-10-000-concurrent-sandboxes-and-5-000-vcpus-per-minute), our Pro plan supports 10,000 concurrent sandboxes and 5,000 vCPU allocated / min with dynamic allocation quotas that ramp up during sustained usage. I'm starting with a default of **250** concurrent sandboxes for the workflow, chosen as a round number similar to our existing 256 matrix job limit, but we can very likely increase this as our benchmark + attempt counts grow. **Note on Sandbox retries** Sandboxes creation is subject to [dynamic quotas](https://vercel.com/docs/limits#dynamic-quotas) on vCPU allocation limits. The Sandbox SDK already implements a [basic retry policy](https://github.com/vercel/sandbox/blob/bf2bc66003fc89cf07a1346a7ea63951747cbec6/packages/vercel-sandbox/src/api-client/with-retry.ts#L55-L71), but it's not the most forgiving, so we add our own exponential backoff retries on top. **Vercel project/creds** Repo is currently wired up to the [evals-runner](https://vercel.com/supabase/evals-runner) project on Vercel via `VERCEL_PROJECT_ID`, `VERCEL_TOKEN`, `VERCEL_TEAM_ID`, a carryover from the Vercel Workflows spike. This project is configured w/ the 3 provider API keys. If desired, we could move sandbox usage back to the evals project since we're not deploying any new application entrypoints with this PR. **Job artifacts** Previously each matrix job in `run-evals` uploaded a separate results artifact which we'd then download and combine. Now, since there's only one `run-evals` job we upload a single combined `raw-results` artifact. <img width="2306" height="540" alt="CleanShot 2026-08-13 at 20 44 50@2x" src="https://github.com/user-attachments/assets/e22bf804-0066-4097-abcc-7cffa9d84330" /> **Job logs** I intentionally don't wait for `command.logs()` because it sometimes loses connection and drops output. Instead, I consume `command.output('both')` so you'll see the `RUN` and `PASS`/`FAIL'` appear only at the end of each sandbox's eval execution, which seems like a reasonable tradeoff. **Sample runs:** - https://github.com/supabase/evals/actions/runs/31542233188 - https://github.com/supabase/evals/actions/runs/31543310069 (tests concurrency queuing) - https://github.com/supabase/evals/actions/runs/31732096448 (codex gpt 5.4 mini, benchmark suite) - https://github.com/supabase/evals/actions/runs/31735507954 (full benchmark refresh) Results look normal after benchmark refresh ([Preview](https://evals-9wrtapw4g-supabase.vercel.app/)): <img width="2374" height="952" alt="CleanShot 2026-08-13 at 17 02 22@2x" src="https://github.com/user-attachments/assets/a49cc514-a421-4c29-b2f5-16e3cb8ef196" /> **How to test** I test by manually dispatching the [refresh workflow](https://github.com/supabase/evals/actions/workflows/eval-refresh.yml) against this branch, though it's a slow / compute heavy thing to refresh the full suite, so you can probably just review the existing runs or dispatch a smaller refresh. **Next steps** Currently, benchmark execution takes ~33 min, comparable to [our latest refresh](https://github.com/supabase/evals/actions/runs/31185759729) on the GitHub Actions matrix. From here, we're planning to parallelize attempts instead of giving agents 2 attempts in sequence. Depending on how many sandboxes this ends up spawning, it'll potentially decrease execution time as attempts can run in parallel. Another optimization I haven't landed here is the warm boot trick from Pedro's #114, I'll likely revisit that as a follow up since it hasn't regressed overall benchmark duration yet. **Related:** - #114 - #186 Closes AI-912 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incomplete proof of concept using Vercel Workflows to orchestrate Sandboxes for running evals outside of GitHub Actions. Workflows can be dispatched via a new Nitro app in
apps/runner, the deployment of which has Deployment Protection so you can use a bypass secret to authenticate.I started with a single workflow
runEvalWorkflowcomposed of 3 steps:createEvalSandboxrunEvalInSandboxdeleteEvalSandboxIdea being that sandbox creation and deletion could fail, so this will retry + continue on step failures as needed, and gracefully rollback sandbox creation on intermediate errors.
A second workflow
runEvalsWorkflow(plural) uses the same steps, but adds loops / retry logic to run a batch of multiple eval runs.I added logic so that agents write their stream of output to the workflow logs (i.e. tool calls, assistant message parts), as well as the normal framework logs (eval run status) to 2 stream "namespaces" for each run.
Not yet included:
Note the majority of this diff is
pnpm-lock.yamlRelated: #114
Ref AI-912