feat: parallel eval attempts - #203
Draft
Rodriguespn wants to merge 3 commits into
Draft
Conversation
Adds --parallel-attempts to the runner: instead of one sandbox running a pair's `runs` attempts sequentially with stop-on-pass, fan every (pair x attempt) out to its own sandbox concurrently and pass the eval if at least one attempt passed. Each attempt is scored exactly as today (--runs 1), so run-eval.ts is unchanged; the any-pass is a pure OR-aggregation over the per-attempt result files (the representative passing tree is kept, attempts set to the number run). Since the bottleneck is agent time per run, parallelizing the attempts can cut wall-clock roughly in half when concurrency covers pairs x runs (one wave). The workflow enables it via vars.EVAL_PARALLEL_ATTEMPTS and sizes concurrency to pairs x runs; default (unset) keeps sequential stop-on-pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Exports aggregateAttempts and adds cases for: passes if any attempt passed (keeps the passing tree), fails only when all failed, counts only attempts that produced a result, and throws when none did. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rodriguespn
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #201: parallel eval attempts (any-pass) — does the run time halve?
Draft, stacked on #201 (warm-boot snapshot). Per the thread conclusion: the snapshot showed the bottleneck is agent time per run, so the next lever is running a pair's attempts in parallel instead of sequentially.
What it does
--parallel-attempts: instead of one sandbox running a pair'srunsattempts sequentially with stop-on-pass, fan every (pair × attempt) out to its own sandbox concurrently. An eval passes if at least one attempt passed (any-pass), matching the agreed success criterion.run-eval.tsis unchanged. Each attempt runs--runs 1and is scored exactly as today; the any-pass is a pure OR-aggregation over the per-attempt result files in the runner (the representative passing tree is kept,attemptsset to the number run). No scoring-semantics change inside the harness.vars.EVAL_PARALLEL_ATTEMPTSand sizes concurrency to pairs × runs so attempts run in one wave. Default (unset) keeps sequential stop-on-pass → no behavior change.The catch (why concurrency matters)
The halving only materializes when concurrency ≥ pairs × runs (≈474), so all attempts run in one wave. At the current 250 it would be ~2 waves = the same wall-clock. The workflow therefore bumps concurrency to pairs × runs for this mode. The real ceiling is AI-provider rate limits at ~474 concurrent agent calls — this run measures both the wall-clock and the pass rate to check the environment didn't cause agents to fail.
Result — ~39% faster (near the floor), fidelity preserved
run-evalsNot quite half, but close to the floor. 28m10s → 17m12s = −39%. The wall-clock floor is the slowest single attempt (828s ≈ 14 min,
build-functions-005); the sequential run was ~2× that (the critical-path eval doing 2 sequential attempts), so the critical path did roughly halve (≈27→14 min). The residual gap from a clean 50% is fixed per-attempt overhead (warm setup + download + aggregation) that doesn't parallelize away.Fidelity holds. 474/474 booted warm, 0 sandbox failures, and 0 evals errored (
💥 ERR). Pass rate 86.4% (190/220) vs 88.2% is run-to-run noise, not throttling: the flips are bidirectional and scattered across every agent/provider (10 PASS→FAIL, 6 FAIL→PASS), whereas throttling would concentrate failures. Attempt distribution: 185 pairs 2/2, 22 pairs 1/2 (any-pass saved these), 30 pairs 0/2.Cost tradeoff: parallel always runs all
runsattempts (474 sandboxes) vs sequential's ~1.1× average — roughly 1.8× the sandbox-compute for the 39% wall-clock win (Pedro's "a lot of compute to keep in mind"). Worth pairing with a lowervcpusand/or excluding the slowest eval from the critical path as follow-ups.Testing
test:vercel-runnergreen.attempts=2, single clean output dir.🤖 Generated with Claude Code