How to Evaluate AI Agents with Harbor

Harbor is an open-source framework for testing and training AI agents with sandboxed tasks. It turns one on 7 November as the task format behind Terminal-Bench 2.0 and, increasingly, the format open reinforcement-learning (RL) trainers use to teach models to work in a terminal. It won by being dull.

TL;DR

• Harbor is an open-source framework and task format from the Terminal-Bench team, launched on 7 November 2025 with Terminal-Bench 2.0.
• A task is a folder: an instruction, a container, a test script that writes a number to a file, and a reference solution.
• A trial is one agent’s attempt at one task; its reward feeds a leaderboard, a fine-tuning dataset or an RL loop.
• Tinker, TRL, SkyRL, Prime RL and NVIDIA NeMo Gym all read the same folders. Write a task once, run it anywhere.
• The format is boring on purpose. The framework is a commodity; the tasks are the asset.

The problem Harbor solves

An agent is a program that gives a model a goal and lets it run commands in a loop until it decides it is done. Claude Code, Codex CLI and OpenHands are agents: the model is the brain, the agent is the hands and the loop. (I have written about why the system around the agent matters more than the agent itself.) A harness is the code around the agent: it sets up the test, starts the agent, grades the result and records everything. A CI runner, but for agents.

Until late 2025 every lab and vendor wrote its own harness, and every task was welded to the harness that ran it. A Terminal-Bench 1.0 task only ran in the Terminal-Bench runner; porting it elsewhere meant porting the grader, the container and the glue. Adithya S K’s Why Harbor exists describes the fix: decouple the task from the harness and write adapters for the agents instead — 26 agent adapters at the time (the docs now list about 45).

That is what the Terminal-Bench team shipped on 7 November 2025: Harbor, built by Laude Institute with Stanford and co-led by Mike Merrill and Alex Shaw. Their FAQ says they rewrote the original evaluation harness from the ground up; Shaw’s launch line, quoted by VentureBeat, was “the package we wish we had had while making Terminal-Bench”. Davy Monte’s explainer has the friendliest one-liner: a testing arena for agents.

A task is a folder

A Harbor task is a unit test for an agent: a prompt, a box to work in, and a script that says pass or fail. The task docs call it “an instruction, environment, and test script”. The folder has five parts.

instruction.md is what the agent reads. In the tutorial task it is three lines: generate an SSH key pair at ~/.ssh/id_rsa, no passphrase.

task.toml holds metadata and limits: author, category, tags, timeouts for the agent, the test script and the build, optional CPU, memory and GPU, and os = "linux" or "windows".

environment/ is the box. A sandbox is an isolated, disposable computer, usually a container, that the agent works in and that is thrown away afterwards: a hotel room, fresh for each guest. Most tasks use a Dockerfile; multi-container setups use docker-compose.yaml; HPC clusters use Apptainer.def.

tests/test.sh is the verifier: a script that runs after the agent stops and decides how well it did. It must write a number, the reward, to /logs/verifier/reward.txt, or to /logs/verifier/reward.json when a task has several named scores. The reward is the grade on the exam. In the tutorial: if pytest passes, write 1; otherwise 0.

solution/solve.sh is the oracle: a known-good solution shipped with the task. Run the task with the oracle as the agent and you prove the task is solvable and the grader works. It is the answer key a teacher checks before printing the exam.

Notice what is missing. Nothing in the folder knows which agent will open it, which model it runs on, or whether the reward feeds a leaderboard or a training loop. An HTTP resource does not know which client will GET it. Same idea.

What happens in a trial

A trial is, in the docs’ words, an agent’s attempt at completing a task. It has four moves.

First, the sandbox starts, on local Docker or a cloud provider; the README lists Daytona, Modal, LangSmith, Blaxel and Novita, which is how one job runs thousands of sandboxes in parallel.

Second, the agent acts. It reads the instruction, runs commands, edits files, and stops when it thinks it is done or the timeout hits. Everything it saw and did is recorded as a trajectory: the full transcript of prompts, commands and outputs. The flight recorder.

Third, the verifier runs. Only now does Harbor copy tests/ into the sandbox and execute test.sh, which writes the reward file.

Fourth, the reward leaves the sandbox, and here three often-confused words separate. An eval runs many trials and reports the average; the model does not change. SFT, supervised fine-tuning, keeps the trajectories that earned a high reward and trains the model to imitate them: studying worked examples. RL, reinforcement learning, runs many trials, scores each one, and nudges the model toward whatever scored higher: practice with a coach who only tells you the score. In RL vocabulary a trial with its trajectory and reward attached is a rollout: one game played to the end, result written on it. Harbor’s RL page says training “requires generating rollouts and recording tokens and rewards”; the trainer reads the reward off the trial result.

Why the format is shaped like this

Most of the design is a list of things that went wrong in Terminal-Bench 1.0, documented on the task-difference page.

Tests leaked into the image. In 1.0 the environment and the tests lived in the same folder, and authors kept copying tests/ into the container by accident, where the agent could read the answers. Now the image is built from environment/ alone and the harness copies tests in at verification time.

Rewards were parsed. A parser inside the benchmark repo turned the test command’s stdout into pass or fail. Now the task writes its own number, and it does not have to be binary: 0.6 is legal, and /logs/verifier/reward.json can carry several named scores.

Solutions were one file; solution/ is now a folder that carries whatever the solve script needs. Config moved from task.yaml to task.toml and the instruction got its own file. Tasks can have steps: a steps/ layout with per-step instruction, tests and solution, for work that must happen in order.

And Windows exists. Set os = "windows" and the task runs in a Windows container, with solve.bat and test.bat writing the same reward file. That is primarily local Docker on a Windows host, though some cloud sandboxes also support Windows.

None of this is clever. Separation of concerns, explicit contracts, no magic parsing. The industry forgets these fundamentals every few years; I wrote about fixing the engine and throwing away the map in September. Harbor’s authors kept the map.

Getting value from this? Subscribe and get the next essay by email.

Who uses it

Terminal-Bench 2.0 is the flagship: 89 tasks, each with several hours of human and LM-assisted validation. Terminal-Bench 2.1 keeps the 89 and patched 26 of them for bugs, timeouts and reward hacking. The hill-climb has been fast: on Benchtalks in March 2026, Shaw put scores at 20 to 30 percent in development, 40 to 45 at launch and around 75 by spring, with every task now solved at least once.

The trainers all read the same folders. Thinking Machines’ Tinker loads a task directory and uses it as an RL environment. Hugging Face TRL wraps a task in a HarborEnv for its RL trainer; the reward call runs the verifier. SkyRL‘s integration was built with the Harbor team; Prime RL, with its Verifiers library, has a cookbook example. NVIDIA’s NeMo Gym runs the 89 Terminal-Bench 2.1 folders for evaluation. Five consumers, zero task rewrites. For a practitioner view of running agent teams day to day, see How I Run a Team of AI Agents with Grok Bot.

The Harbor Cookbook, released 27 March 2026, shows the range: a simple task, a multi-container task where the agent talks to a local REST API, MCP tools from a FastMCP server, a simulated user the agent must interview, DNS blacklisting, and skills. Harbor’s task format also supports ordered multi-step tasks. The repo adds RL examples for Tinker, Prime RL plus Verifiers and SkyRL, and harness optimisation with GEPA.

The wider list is in awesome-harbor: SWE-bench Verified ported to the format, Alibaba’s terminal-bench-pro, NVIDIA’s synthetic terminal tasks, the SWE-gen datasets.

Try it in 10 minutes

uv tool install harbor                  # or: pip install harbor
harbor task init my-task                # scaffolds the folder above
# edit instruction.md, environment/Dockerfile, tests/test.sh, solution/solve.sh
harbor run -p my-task -a oracle         # the answer key must score 1
harbor run -d terminal-bench@2.0 -a claude-code -m <model> --n-concurrent 4

The last line downloads the benchmark and runs a real agent against it, four sandboxes at a time, on local Docker. Check the docs before copying flags: Harbor ships a dev build almost daily and a numbered release every few weeks, so the syntax moves.

What Harbor is not

It is not a trainer. Harbor produces rollouts and rewards; the gradient updates happen in Tinker, TRL, SkyRL or Prime RL.

It is not a GUI or computer-use framework by itself. The core is a terminal in a container; the cookbook has desktop recipes for Ubuntu and Windows, but you bring the desktop.

There is no macOS task OS: Linux containers everywhere, Windows containers on a Windows host. Harbor itself runs fine on Macs.

Graders are shell tests. If you can score it from test.sh, with pytest, a curl or a diff, it works. If you cannot, you call a model inside the script, and then you have a judge problem, not a Harbor problem.

And tasks still need humans. Each 2.0 task took hours of validation; 2.1 still had to patch 26. On Benchtalks, Shaw said 89 tasks are not enough and the field needs a thousand times more. Harbor is the factory. It does not design the parts.

The scarce thing is the task

One year in, the format is plumbing. Five frameworks and a dozen benchmarks read the same folder; all they agreed on was a file name and a path. That is a good contract: small enough to be boring, stable enough to build on. Web APIs converged the same way once they stopped inventing verbs and used HTTP’s.

So the value has moved. Nobody wins by writing a better runner; it is replaceable by design. The scarce thing is a well-specified, verifiable task: an instruction a human would accept, an environment that does not leak the answer, a grader that cannot be gamed. Those take hours each, and the field needs orders of magnitude more. Do not build a framework. Write a folder.

Enjoyed this? Get one or two essays a month — subscribe.