Skip to content

Port ARC-AGI-3 harness runtime seam to Codex (Codex CLI, AGENTS.md, token accounting) - #3

Closed
dastin359 wants to merge 9 commits into
claude/athanor-cc-harness-variant-jpqw7tfrom
codex/port-arc-agi-3-harness-logic
Closed

Port ARC-AGI-3 harness runtime seam to Codex (Codex CLI, AGENTS.md, token accounting)#3
dastin359 wants to merge 9 commits into
claude/athanor-cc-harness-variant-jpqw7tfrom
codex/port-arc-agi-3-harness-logic

Conversation

@dastin359

Copy link
Copy Markdown
Owner

Motivation

  • Replace the Claude-Code-specific session/launch semantics with a Codex-compatible runtime so the ARC-AGI-3 harness can run under codex exec while preserving ledger-first scoring and clean-rollout policy.
  • Preserve the five domain modules (grids.py, ledger.py, rules.py, scoring.py, client.py) unchanged so audited scoring and ledger invariants remain authoritative.
  • Reimplement the runtime seam (workspace template, CLI invocation, stream parsing, nudging/resume policy) to use Codex-native JSONL events and sandbox semantics rather than Claude CLI flags.

Description

  • Rewrote the runtime seam in src/athanor/ccarc3/session.py to: pin a Codex model (DEFAULT_MODEL = "gpt-5.3-codex"), build non-interactive invocations via codex exec --json, emit a sandbox and a model_reasoning_effort setting, and capture/resume the authoritative Codex thread id from thread.started events instead of assigning a session id pre-launch.
  • Switched solver-facing workspace surface from CLAUDE.md to AGENTS.md and updated the workspace generation template and ablation tooling to redact baselines from the new AGENTS.md while preserving fenced code blocks.
  • Ported stream/telemetry handling to Codex semantics: run_cost now reads turn.completed / usage counters and records input_tokens, cached_input_tokens, and output_tokens instead of fabricating Claude-only total_cost_usd/duration_ms. The nudge/resume flow now uses codex exec resume with the captured thread id when possible.
  • Updated operational tooling and audit/proofread logic to the Codex JSONL shape and to the new evidence/scratchpad defaults, including tools/proofread_trace.py (item-oriented parsing), tools/clean_rollouts.py, supervisor/heartbeat scripts, tools/* scratch defaults, and tests that reference the runtime seam or evidence filenames.

Testing

  • Ran targeted ARC-AGI-3 test sets; selected test runs against the modified modules completed cleanly: pytest on the focused ccarc3-related tests produced a green result for the exercised suite (final observed run: 486 passed, 5 skipped).
  • Exercised the session/unit tests: pytest tests/test_ccarc3_session.py and related CCARC3 unit tests ran and were updated to reflect Codex CLI and AGENTS.md changes (these tests passed in the runs performed).
  • Ran the mutation battery on the audited domain modules; the mutation runner exercised the grids, rules and ledger mutants and observed the expected catches (no unexpected survivors reported for the audited mutants before the run was interrupted for time budget reasons).
  • Compiled and smoke-tested tools and scripts with python -m compileall and multiple pytest batches; the modified ccarc3 surface and proofreader were verified to parse Codex JSONL events and to preserve the ledger/scoring invariants during the automated checks.

Codex Task

@dastin359
dastin359 deleted the branch claude/athanor-cc-harness-variant-jpqw7t August 25, 2026 22:05
@dastin359 dastin359 closed this Aug 25, 2026
@dastin359
dastin359 deleted the codex/port-arc-agi-3-harness-logic branch August 25, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant