Add harbor_env — serve Harbor task directories as an OpenEnv environment

[thegovind]

Summary

A new environment, envs/harbor_env, that runs any Harbor task directory as a Gymnasium-style OpenEnv environment.

The task directory is the interface. A directory that harbor run accepts is served here unchanged — no conversion step, no re-authoring, no second copy of the data, and no producer-specific code in OpenEnv. That covers Terminal-Bench-lineage tasks and everything Repo2RLEnv generates from a GitHub repository, so a task set built for evaluation is also a training environment.

# serve the bundled example task
uv run --project envs/harbor_env server

# or point it at your own task set — a directory or a Hub dataset
HARBOR_TASKS_DIR=./tasks HARBOR_MODE=docker uv run --project envs/harbor_env server
HARBOR_TASKS_DIR=hf://datasets/my-org/click-tasks uv run --project envs/harbor_env server

---

Where this sits

Harbor and OpenEnv are not competitors — they sit at different layers, and they agree on the one contract that matters: the reward is produced inside the environment and only forwarded.

flowchart LR
    P["Task producers<br/>Repo2RLEnv · Terminal-Bench<br/>· hand-written"]
    T["Harbor task directory<br/><i>the format</i>"]
    H["harbor run<br/><i>batch evaluation</i>"]
    E["envs/harbor_env<br/><b>NEW</b> — episode loop"]
    TR["Trainer / RL loop<br/>reset · step · state"]

    P -->|emit| T
    T --> H
    T -->|"served unchanged"| E
    E --> TR

    style E fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
    style T fill:#e3f2fd,stroke:#1565c0

Loading

	Harbor	OpenEnv
Kind of thing	A file format + a batch runner	A serving standard + runtime
Unit	A task directory on disk	A running service
Interaction	One-shot: drop agent in sandbox, verify at the end	Stepwise episode loop, resettable by the trainer
Reward	tests/test.sh → /logs/verifier	Observation.reward, produced inside the env

harbor_env is the bridge, and it generalizes the existing envs/tbench2_env/ pattern. The one real behavioural difference: tbench2 derives pass/fail from the verifier's exit code, while Harbor rewards are graded floats, so harbor_env reads the value the verifier wrote.

---

An episode

sequenceDiagram
    participant TR as Trainer <br/>(infrastructure)
    participant E as HarborEnvironment
    participant S as Sandbox <br/>(local or docker)
    participant V as tests/test.sh

    TR->>E: reset(task_id="fix-sum-bug")
    E->>S: boot backend, seed working directory
    E-->>TR: Observation(instruction=instruction.md)

    loop agent works
        TR->>E: step(exec / read / write)
        E->>S: run in the working directory
        S-->>E: output
    end

    TR->>E: step(evaluate)
    E->>S: clear /logs/verifier, stage tests/ -> /tests
    E->>V: bash /tests/test.sh
    V-->>S: reward.json / reward.txt
    E-->>TR: Observation(reward=0.83, done=True)

    Note over E,V: reward.json first, then reward.txt —<br/>exactly Harbor's precedence
    Note over TR,E: evaluate + solve are infrastructure-only,<br/>never in the agent's action space

Loading

Invariants held

• Agents cannot reset. reset / step / state stay on the infrastructure side. evaluate and solve are orchestration controls, not agent actions — an agent that could grade on demand could end its own episode, and one that could solve could hand itself the answer.
• Rewards live inside the environment. harbor_env forwards what tests/test.sh wrote and never computes a reward. If the verifier wrote no reward file, the result is reward=None plus an explicit error — not a 0.0. A fabricated zero is indistinguishable from a genuine failure and would poison a training run.
• Client/server separation. client.py imports nothing from server/.
• The agent cannot reach the answer. read and write go through resolve_within(), confined to the working directory, so /tests and /solution are unreachable. /logs/verifier is wiped immediately before the verifier runs, so a reward file planted during exec cannot survive into the score.

---

Two execution modes

flowchart TD
    A["reset(task_id)"] --> B{"Where is the task's<br/>starting state?"}
    B -->|"ships environment/ seed files<br/><i>self-contained</i>"| C["local mode OK<br/><i>subprocesses, no Docker —<br/>works on HF Spaces</i>"]
    B -->|"Dockerfile / compose /<br/>docker_image"| D{"HARBOR_MODE"}
    D -->|docker| E["run inside the task's<br/>own image"]
    D -->|local| F["refuse with an actionable message<br/><i>rather than grading an empty directory</i>"]

    style C fill:#e8f5e9,stroke:#2e7d32
    style E fill:#e8f5e9,stroke:#2e7d32
    style F fill:#ffebee,stroke:#c62828

Loading

task 'pallets__click-2951' keeps its starting state in a container image
(environment/Dockerfile), which the local backend cannot reproduce.
Run the server with HARBOR_MODE=docker, or use a task that ships its files in environment/.

local mode is documented as a filesystem boundary, not a security boundary — exec runs as the server's own user with the server's own environment. Serve task sets you trust there; use docker for anything else.

---

Verification

The interop claim is the whole point of this PR, so it was measured rather than asserted.

The bundled examples/tasks/fix-sum-bug task is written to run under every runtime (it prefers $HARBOR_* variables and falls back to Harbor's absolute paths). It grades identically in all three:

Runtime	Untouched task	After solution/solve.sh
harbor run -a nop / -a oracle (harbor 0.20.0)	0.600	1.000
harbor_env local mode	0.600	1.000
harbor_env docker mode	0.600	1.000

Reproduce row one with harbor run -p envs/harbor_env/examples/tasks/fix-sum-bug -a nop, and the others with examples/validate_taskset.py.

Separately, a real Repo2RLEnv-emitted pr_runtime task — produced by repo2rlenv's own emitter, so version = "1.0" rather than Harbor's schema_version, plus [metadata.repo2env], WORKDIR /workspace, and a tests/test.sh writing /logs/verifier/reward.txt — scored 0.0 for a no-op agent and 1.0 for the oracle under both harbor run and harbor_env docker mode, with reward-details.json surfacing intact as observation.info["reward_details"].

---

Note

High Risk
New environment runs attacker-controlled shell in local or Docker sandboxes and forwards training rewards; mis-grading or weak isolation could affect RL data and host security.

Overview
Introduces envs/harbor_env, a new OpenEnv environment that serves Harbor task directories unchanged (including Repo2RLEnv output) over the standard reset/step API, with a FastAPI/WebSocket server, HarborEnv client, and wire types for exec / read / write plus infrastructure-only evaluate and solve.

Execution is split between docker (task images, policy enforcement) and local (subprocesses under a per-episode tree for Spaces); local mode refuses tasks that need images or policies it cannot enforce. Rewards are read from verifier artifacts (reward.json then reward.txt, with reward-details.json in info); missing files yield reward=None, not 0.0. Sandboxing limits agent I/O to the workdir, stages /tests only at verify time, and clears verifier logs before grading.

Also adds a bundled fix-sum-bug example task, quickstart / validate_taskset scripts, Docker image defaults, and docs entries (environments/harbor, catalog card, toctree).

^{Reviewed by Cursor Bugbot for commit 5079a77. Bugbot is set up for automated code reviews on this repo. Configure here.}