feat: move eval execution from GitHub Actions to Vercel Sandboxes - #192

Merged

mattrossman merged 42 commits into

mainfrom

mattrossman/ai-912-spike-vercel-sandbox-runner-for-evals

Aug 14, 2026

Commits on Aug 11, 2026

Commits on Aug 13, 2026

Merge remote-tracking branch 'origin/main' into mattrossman/ai-912-spike-vercel-sandbox-runner-for-evals
mattrossman
committed
654af9a
refactor: simplify sandbox retries, share CLI parsing, validate with zod
mattrossman
committed
14ca377
refactor: retry Sandbox.create() through vCPU rate limit, surface API errors
mattrossman
committed
08216de
fix: surface real Sandbox API error details in retry logs
mattrossman
committed
8b6f314
fix: remove noisy setup-step log streaming, report pass/fail live per pair
mattrossman
committed
2d184fa
fix: stream run-eval's own PASS/FAIL output, disambiguate controller labels
mattrossman
committed
fdbb297
fix: tighten per-step timeouts so hangs fail fast instead of riding the session ceiling
mattrossman
committed
8ff355a
chore: refresh eval results
github-actions[bot]
committed
a0dc93d
fix: replace racy live log streaming with post-completion output fetch
mattrossman
committed
7c68bfe
Merge remote-tracking branch 'origin/mattrossman/ai-912-spike-vercel-sandbox-runner-for-evals' into mattrossman/ai-912-spike-vercel-sandbox-runner-for-evals
mattrossman
committed
bded28f
style: fix formatting
mattrossman
committed
6dcae56
fix: drop redundant per-step log lines, keep only the run-eval marker
mattrossman
committed
b78c513
chore: refresh eval results
github-actions[bot]
committed
aaf63d0
chore: refresh eval results
github-actions[bot]
committed
d9afed8
chore: duplicate comment
mattrossman
committed
d70ad06
fix: dedup redundant Vercel credential validation
mattrossman
committed
31e2081
fix: raise default sandbox_concurrency from 4 to 250
mattrossman
committed
501f73a

Commits on Aug 14, 2026

fix: jitter sandbox creation retries, comments
mattrossman
committed
4ce9022
chore: download into existing `results/` dir
mattrossman
committed
b025336
chore: format
mattrossman
committed
6e9a634
fix: results path in workflow
mattrossman
committed
4132dfb
fix: overwrite raw-results artifact so job re-runs can publish
mattrossman
committed
36d2296
fix: raise eval command timeout buffer from 5 to 25 min
mattrossman
committed
f7d766e
fix: validate numeric CLI flags in run-eval, reject empty inline flag values
mattrossman
committed
17f8691
fix: install pnpm version pinned by packageManager in sandboxes
mattrossman
committed
8293d26
fix: match Vercel SDK retry policy in sandbox creation retries
mattrossman
committed
8a288da
chore: format
mattrossman
committed
64436aa
chore: restore results from main
mattrossman
committed
ccc6695
chore: revert newline
mattrossman
committed
d88c44d
fix: only retry known-transient sandbox create errors
mattrossman
committed
5b11a9b
fix: stop outer pair retry from re-attempting terminal sandbox create errors
mattrossman
committed
8fa6fb3
fix: drop unused OIDC credential fallback for Vercel Sandbox auth
mattrossman
committed
6d74fa1
ci: bound run-evals job timeout, publish partial eval results loudly
mattrossman
committed
e0d71d4
test: temporary artificial delay for workflow testing, DO NOT MERGE
mattrossman
committed
489b380
fix: tolerate a missing raw-results artifact when every pair fails
mattrossman
committed
2ec6aff
fix: don't claim results were published when zero pairs finished
mattrossman
committed
ef3d5e1
revert: drop temporary artificial delay used for workflow testing
mattrossman
committed
d07cced
chore: trim comments
mattrossman
committed
03cae1a