A benchmark score is a claim. Harbor makes it very easy to produce one. It doesn’t make it easy to know whether the claim is true.

Harbor has quickly become a default way to run agent benchmarks. It’s the official harness for Terminal-Bench 2.0, and it can run SWE-bench and a long list of other datasets too. I wanted to know where its numbers can mislead you, so I built a small task and tried to break it.

This isn’t a setup guide. The Harbor docs cover installation well, and running Terminal-Bench 2.0 is a single command. What the docs can’t tell you is when the number on your screen is wrong. That’s what this post is about.

Everything here runs on a laptop with Docker. No API keys and no model calls. Every command and output below is real, from Harbor 0.24.0, and where it helps I point to the same failure in a well-known public benchmark. You don’t need to have used Harbor before. I explain each piece as it comes up, and there’s a short list of terms right after the next section.

All the tasks, agents and scripts are in this repo, with a run_all.sh that reproduces every result in the post. Run the commands below from the repo root.

Harbor in two minutes

Harbor comes from the team behind Terminal-Bench and is the official harness for Terminal-Bench 2.0. The vocabulary is small. A task is a directory with an instruction, an environment and a test script. A trial is one agent’s attempt at one task. A job is a batch of trials that Harbor runs in parallel, locally in Docker or on cloud sandboxes like Daytona and Modal.

Figure 1. The life of one trial. The two boxes at the bottom are where scores quietly go wrong. (Original diagram.)

The right half of that picture is what this post is about. The verifier is just a shell script. Its only contract with Harbor is a number written to /logs/verifier/reward.txt, or a JSON object in reward.json. Every number Harbor shows you is built on top of that file.

Terms you’ll see in this post

• Benchmark: a set of tasks plus a way to score them, used to compare agents.
• Agent: the program being tested. Usually a language model that can run commands. In this post, mostly tiny scripts.
• Task: one problem. It has an instruction, a container to work in, and a test.
• Trial: one attempt by one agent at one task.
• Job: a batch of trials run together, for example every task five times.
• Verifier: the test script that checks the agent’s work after it stops.
• Reward: the number the verifier writes. Here 1 means passed and 0 means failed.
• Score: the summary over many trials, usually the mean reward.
• Baseline: an agent so simple that you know its score before you run it. It checks the benchmark, not the agent.

The test subject: a CSV dedupe task

The container has a small customers file with a few duplicate emails. The duplicates use different casing, and one has a trailing space. The agent has to write a clean copy. Harbor scaffolds the task for you:

harbor task init demo/csv-dedupe --no-pytest
csv-dedupe/
├── instruction.md
├── task.toml
├── environment/
│   ├── Dockerfile
│   └── data/customers.csv
├── solution/
│   └── solve.sh
└── tests/
    ├── test.sh
    └── test_outputs.py

The instruction is short and names exact paths, which the Harbor docs recommend so the instruction and the tests agree on what “done” means:

`/app/data/customers.csv` has duplicate customers. Two rows are the same
customer when their `email` values match after trimming whitespace and
lowercasing.
Write a deduplicated copy to `/app/out/customers_clean.csv`:
- keep the header and the original column order
- keep the first occurrence of each customer, in the original row order
- do not modify `/app/data/customers.csv`

The test is plain Python with no dependencies. It checks the header and the exact list of surviving IDs:

import csv, sys
OUT = "/app/out/customers_clean.csv"
EXPECTED_IDS = ["1", "2", "4", "6"]
def main() -> int:
    try:
        with open(OUT, newline="") as f:
            rows = list(csv.DictReader(f))
            header = rows and list(rows[0].keys())
    except FileNotFoundError:
        print(f"FAIL: {OUT} not found")
        return 1
    if header != ["id", "name", "email", "signup_date"]:
        print(f"FAIL: header is {header}")
        return 1
    ids = [r["id"] for r in rows]
    if ids != EXPECTED_IDS:
        print(f"FAIL: ids {ids} != {EXPECTED_IDS}")
        return 1
    print("PASS")
    return 0
sys.exit(main())

Check 1: the oracle must pass and nop must fail

Harbor ships two agents that never call a model, and they’re the cheapest checks you can run on a task.

• oracle is Harbor’s name for the reference answer. It runs the task’s solution/solve.sh, the script the task-author wrote to prove the task can be solved. If the oracle fails, the task itself is broken: the environment, the tests or the solution.
• nop (short for “no operation”) does nothing. Harbor starts the container, the agent touches nothing, and the verifier runs straight away. If nop passes, your tests are checking something that was already true before any work happened.

Together they pin down both ends. A correct solution scores 1, and doing nothing scores 0. Every real agent should land somewhere in between.

This isn’t my idea. The Terminal-Bench 2.0 authors use the same pair as an automated gate on every task: the oracle solution has to pass and a dummy solution has to fail.

harbor run -p csv-dedupe -a oracle
harbor run -p csv-dedupe -a nop

-p is the path to the task folder and -a picks the agent. The oracle came back with a mean of 1.000, a perfect score. My first nop run came back like this:

adhoc • nop
┏━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┓
┃ Trials ┃ Exceptions ┃  Mean ┃
┡━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━┩
│      0 │          1 │ 0.000 │
└────────┴────────────┴───────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Exception               ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ RewardFileNotFoundError │     1 │
└─────────────────────────┴───────┘

Harbor prints this table at the end of every run. Trials counts the trials that finished and got a reward from the verifier. Exceptions counts the trials where something raised an error along the way. Mean is the average reward, from 0 to 1. The second table names each error and how often it happened.

A mean of 0 is what I wanted. But look at the other two numbers. Zero completed trials, one exception. The agent didn’t fail. The trial crashed.

Check 2: make a failing test write a 0

My test.sh followed the verifier example in the Harbor docs:

#!/bin/bash
set -euo pipefail
python3 /tests/test_outputs.py
if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

If you don’t write much bash, here’s what you need to read this script. Every command ends with an exit code: 0 means it worked, and anything else means it failed. My Python test exits with 0 when the output is right and 1 when it isn’t. $? holds the exit code of the last command. set -e tells bash to stop the whole script as soon as any command fails. (The -uo pipefail part adds two related safety checks that don’t matter here.)

That’s the bug. When the test fails, it exits with 1. Because of set -e, bash stops the whole script on that line. The if never runs, so nothing is written to reward.txt. The $? check can only ever see a 0. (In the repo, this version lives in csv-dedupe-buggy/.)

Harbor then raises RewardFileNotFoundError. This is the part that matters for scores: the built-in Mean metric treats a trial with no reward as 0 (see metrics/base.py). So a broken verifier and a failing agent produce the same headline number. The difference only shows up in the exceptions column.

It also survives review. On a passing run the script works fine, so the oracle check is green and nobody looks twice. Here’s the fixed version:

#!/bin/bash
# No `set -e`: a failing test must still write a reward.
mkdir -p /logs/verifier
if python3 /tests/test_outputs.py; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

The fix drops set -e and puts the test command straight into the if. Bash runs the test, and the if picks a branch based on whether it passed. Either way, a reward gets written. Same nop run, fixed verifier: 1 trial, 0 exceptions, mean 0.000. That’s a real zero.

One more trap in the same file. A verifier can report more than one score, for example one for correctness and one for speed. In that case it writes them as a JSON object to reward.json, like {"correct": 1, "fast": 0}. The test.sh that harbor task init generates says these go to rewards.json, with an s. Harbor reads reward.json, singular. If you follow the comment, you get the same RewardFileNotFoundError.

A crash is the loud version. A verifier can also be wrong quietly, by rejecting work that is actually correct, and that looks exactly like an agent failure. This happens in the big public benchmarks too. SWE-bench is one of the best known. It gives an agent a real GitHub issue from a Python project and asks it to fix the code. The unit tests that came with the real fix decide whether the agent passed. When OpenAI built SWE-bench Verified in 2024, a cleaned-up subset meant to drop the bad tasks, annotators reviewed 1,699 SWE-bench tasks and flagged 61.1% of them for unit tests that may reject valid solutions. Only 500 tasks made it into the final set.

Even that cleaned-up set didn’t hold. In February 2026 OpenAI stopped reporting SWE-bench Verified. In an audit of 138 problems that its o3 model didn’t solve reliably, 59.4% had material issues with the tests or the description, and 35.5% had tests narrow enough to reject functionally correct fixes.

The oracle check can’t catch this, because the oracle solution is the one the tests were written for. The only defence I know is to read a sample of failed trials yourself. harbor view jobs opens a local viewer for each trial. It shows the agent’s trajectory, which is the step-by-step record of every command the agent ran and what came back, next to the verifier’s output. That makes the reading a lot less painful.

Check 3: add a lazy baseline

Oracle and nop only prove the two ends. They don’t prove the verifier checks the actual work. For that I add a third baseline. A baseline is an agent so simple that you know what it should score before you run it. If it scores something else, the problem is in the task, not the agent. This one does the cheapest thing that might pass.

A custom agent in Harbor is a small Python class. This one ignores the instruction and copies the input file to the output path:

# agents/lazy.py
from harbor.agents.base import BaseAgent
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext
class LazyCopyAgent(BaseAgent):
    """Ignores the instruction and copies the input file to the output path."""
    @staticmethod
    def name() -> str:
        return "lazy-copy"
    def version(self) -> str:
        return "0.1.0"
    async def setup(self, environment: BaseEnvironment) -> None:
        pass
    async def run(self, instruction: str, environment: BaseEnvironment,
                  context: AgentContext) -> None:
        await environment.exec(
            command="mkdir -p /app/out && "
            "cp /app/data/customers.csv /app/out/customers_clean.csv"
        )

Here’s what each part does. name() and version() are labels that Harbor puts in the results. setup() runs once before the attempt, which is where a real agent would install its tools. This one needs nothing. run() is the agent itself: Harbor hands it the instruction and the task’s container. environment.exec() runs a shell command inside that container, the same way an agent built on a model runs the commands the model picks.

Built-in agents like oracle have short names. For your own agent you tell Harbor where the class lives: the module path (agents.lazy means the file agents/lazy.py), a colon, then the class name. PYTHONPATH=. tells Python to look for that module in the current folder:

PYTHONPATH=. harbor run -p csv-dedupe -a agents.lazy:LazyCopyAgent

I ran all three baselines (oracle, nop and the lazy copy) against my real verifier, and against a weak one that only checks that the output file exists and has the right header. The weak one is csv-dedupe-weak/ in the repo.

Figure 2. Results from six real runs. The weak verifier passes the oracle and nop gate, and still gives a perfect score to an agent that ignored the instruction. (Original diagram.)

That’s the uncomfortable part. The weak verifier looks fine by the standard check. Real agents find gaps like this one too. It’s called reward hacking: the agent gets the reward without doing the task, usually by exploiting something the test doesn’t check. I’ll get into it properly in a later post. A lazy baseline is a cheap way to catch the obvious gaps before a model does.

To write one, look at your task and ask what a shortcut would look like. Copy the input. Print the expected string. Edit the test file. Read the git history. Then make one tiny agent per shortcut. Each one is about 25 lines.

That last shortcut isn’t hypothetical. Each SWE-bench task starts from a copy of the project’s repository at the commit just before the real fix. In September 2025, a GitHub issue on the SWE-bench repo showed that the task images still carried the repository’s later history, so the real fix was sitting in the git log, one command away. Agents found it: runs with Claude 4 Sonnet, GLM-4.5 and Qwen3-Coder used commands like git log --all to look at the future commits that contained the fix. The maintainers estimated only a handful of trajectories were affected, and swebench 4.1.0 fixed it by stripping all history after each task’s base commit. Still, a lazy baseline that runs git log --all and greps for the issue number would have flagged the hole on day one.

Check 4: an exception can still be a pass

Harbor’s summary table puts an Exceptions count next to the Mean. It’s tempting to read Exceptions as “infra problems” and Mean as “agent quality”. Neither reading is quite right.

I wrote an agent that writes the correct output and then hangs, like a real agent that finishes the work but never leaves its loop. Every task sets a time limit for the agent in task.toml (timeout_sec = 120.0 in mine), and --agent-timeout-multiplier scales that limit for one run. A multiplier of 0.05 cuts it to 6 seconds:

PYTHONPATH=. harbor run -p csv-dedupe -a agents.hang:SolveThenHangAgent \
    --agent-timeout-multiplier 0.05

adhoc • solve-then-hang
┏━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┓
┃ Trials ┃ Exceptions ┃  Mean ┃
┡━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━┩
│      1 │          1 │ 1.000 │
└────────┴────────────┴───────┘
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Exception         ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ AgentTimeoutError │     1 │
└───────────────────┴───────┘

After an agent timeout, Harbor records the error and still runs the verifier. If the work is on disk, it counts. I think that’s the right call for scoring. It does mean one Exceptions number mixes trials that passed late, trials whose verifier broke, and trials where the container itself fell over.

Retries need the same care. With -r (--max-retries), Harbor runs a crashed trial again. It’s off by default. Even when you turn it on, Harbor won’t retry agent timeouts or verifier errors like the missing reward file, because a second attempt would hide a slow agent or a broken verifier. The full list is in RetryConfig. Other errors, such as a container that fails to start, do get retried, and the job’s result.json records how many retries happened. Check that number before you compare two runs.

So before I trust a mean, I sort every trial into a bucket. Harbor writes a result.json into every trial’s folder. Here is the one from the timeout run above, trimmed to the fields that matter:

{
  "task_name": "demo/csv-dedupe",
  "verifier_result": {
    "rewards": { "reward": 1.0 }
  },
  "exception_info": {
    "exception_type": "AgentTimeoutError",
    "exception_message": "Agent execution timed out after 6.0 seconds"
  }
}

Two questions are enough to sort any trial. Did the verifier produce a reward (rewards)? And did anything raise an error along the way (exception_info)? This trial says yes to both: it timed out, and the work still passed.

Figure 3. Four buckets that the single Exceptions column hides. (Original diagram.)

Here is a script for triage:

# scripts/triage.py
import json, sys
from collections import Counter
from pathlib import Path
VERIFIER_BROKEN = {"RewardFileNotFoundError", "RewardFileEmptyError",
                   "VerifierOutputParseError", "VerifierTimeoutError"}
def bucket(r: dict) -> str:
    exc = (r.get("exception_info") or {}).get("exception_type")
    rewards = (r.get("verifier_result") or {}).get("rewards")
    if rewards is None:
        return "verifier broken" if exc in VERIFIER_BROKEN else f"infra: {exc}"
    passed = next(iter(rewards.values())) == 1
    if exc == "AgentTimeoutError":
        return "timed out, passed" if passed else "timed out, failed"
    return "passed" if passed else "failed"
job = Path(sys.argv[1])
counts = Counter(bucket(json.loads(p.read_text()))
                 for p in job.glob("*/result.json"))
for name, n in counts.most_common():
    print(f"{n:4d}  {name}")

$ python3 scripts/triage.py jobs/buggy-nop
   1  verifier broken
$ python3 scripts/triage.py jobs/timeout-test
   1  timed out, passed

On a real job with hundreds of trials, this split is the first thing to look at. The “verifier broken” and “infra” buckets point at the task or at the machine running it, so they say nothing about the agent.

Check 5: one run is an anecdote

Agents built on language models are stochastic: the model samples its answer, so the same agent on the same task can succeed once and fail the next time. One pass tells you the agent can solve a task, not that it will. To see how much that matters, I gave the toy task 19 siblings with different data, so 20 tasks in total, and wrote an agent with a known skill level. It solves a task 60% of the time and does nothing otherwise. Real agents are messier, but here at least I know the right answer.

# agents/flaky.py (setup, name and version as in LazyCopyAgent)
class CoinFlipAgent(BaseAgent):
    """Solves the task with probability 0.6, otherwise does nothing."""
    async def run(self, instruction, environment, context) -> None:
        if random.random() < 0.6:
            script = SOLVE.read_text()  # the task's solution/solve.sh
            await environment.exec(command=f"bash <<'EOF'\n{script}\nEOF")

Then I ran it five times, one attempt per task each time:

python3 scripts/make_dedupe20.py      # writes the 20 tasks to dedupe-20/
PYTHONPATH=. harbor run -p dedupe-20 -a agents.flaky:CoinFlipAgent

Figure 4. Five Harbor runs of an agent whose true solve rate is 0.60, on the same 20 tasks. (Original chart, real runs.)

Same agent, same tasks, and the score moved anywhere from 0.45 to 0.65. If two different agents showed a 20-point gap, you’d call it a clear win. Here it’s pure luck. Terminal-Bench 2.0 has 89 tasks, which shrinks the spread, but it doesn’t make it go away.

The fix is more attempts per task. -k 5 runs every task five times, so 20 tasks become 100 trials:

PYTHONPATH=. harbor run -p dedupe-20 -a agents.flaky:CoinFlipAgent -k 5
dedupe-20 • coin-flip
┏━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Trials ┃ Exceptions ┃  Mean ┃ Pass@2 ┃ Pass@4 ┃ Pass@5 ┃
┡━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│    100 │          0 │ 0.560 │  0.805 │  0.930 │  0.950 │
└────────┴────────────┴───────┴────────┴────────┴────────┘

With 100 trials the mean lands at 0.56, much closer to the true 0.60. Harbor also reports pass@k. Pass@k is the chance that at least one of k attempts at a task succeeds, averaged over all tasks. So Pass@2 = 0.805 means: give the agent two tries at a task, and about 80% of the time at least one of them works. Harbor estimates it from your attempts with the formula from the Codex paper (Chen et al., 2021). It only shows k values that are powers of two or multiples of five, up to your attempt count, which is why a -k 3 run only shows Pass@2.

Be careful which of those numbers you quote. Pass@5 says 0.95. So I counted how many tasks the agent solved in all five attempts. The answer was zero, out of 20.

Figure 5. The same 100 trials, read two ways. Pass@k rewards an agent for getting it right once. Solving 5 of 5 asks whether you can rely on it. (Original chart, real runs.)

Pass@k answers “can it ever do this?”. For an agent you’re going to ship, the better question is usually “will it do this every time?”. That one is sometimes written as pass^k: the chance that all k attempts succeed. Harbor doesn’t report it, but it’s a few lines over the trial results:

# scripts/always.py
import json, sys
from collections import defaultdict
from pathlib import Path
by_task = defaultdict(list)
for p in Path(sys.argv[1]).glob("*/result.json"):
    r = json.loads(p.read_text())
    rewards = (r["verifier_result"] or {}).get("rewards") or {}
    by_task[r["task_name"]].append(rewards.get("reward", 0))
always = sum(all(x == 1 for x in v) for v in by_task.values())
print(f"solved every time: {always}/{len(by_task)}")

$ python3 scripts/always.py jobs/coin-k5
solved every time: 0/20

For comparison, the Terminal-Bench 2.0 authors ran every model and agent pair at least five times, 32,155 trials in total, and report 95% confidence intervals. A confidence interval is the range the true score probably sits in, given how much the runs moved around. When two agents’ ranges overlap, the gap between them may just be luck. That’s the bar I’d hold any comparison to.

What all those attempts cost

Every attempt is a full trial: a fresh container, the agent, the verifier and teardown. Even with no model in the loop, that adds up. Here is the oracle on the 20 tasks, on the 2-vCPU cloud VM I used for this post. -n sets how many trials Harbor runs at the same time:

Run one at a time, each trial took about 14 seconds. The agent and the verifier together needed less than 1.5 seconds of that. The rest is Harbor building, starting and stopping containers. Concurrency helps until you run out of cores, so going from 4 to 8 on two vCPUs only saved another 19 seconds.

With a real model, the overhead stops mattering and the agent dominates. The Terminal-Bench 2.0 paper reports that most trials finish in under 20 minutes, but some run up to two hours and burn close to 100 million tokens on a single task. Tokens are the chunks of text a model reads and writes, and API pricing is per token. At five attempts on 89 tasks, that’s 445 trials per agent. Harbor records n_input_tokens, n_output_tokens and cost_usd per trial when the agent integration reports them, so I put cost per solved trial right next to the mean:

# scripts/cost.py
import json, sys
from pathlib import Path
trials = [json.loads(p.read_text())
          for p in Path(sys.argv[1]).glob("*/result.json")]
cost = sum((t["agent_result"] or {}).get("cost_usd") or 0 for t in trials)
solved = sum(
    1 for t in trials
    if ((t["verifier_result"] or {}).get("rewards") or {}).get("reward") == 1
)
print(f"{len(trials)} trials, {solved} solved, ${cost:.2f} total")
print(f"${cost / max(solved, 1):.2f} per solved trial")

Two smaller things that cost me time

Job names are resume keys. Harbor uses the job name to find the job’s folder, so it can pick up a run that stopped halfway. If you pass --job-name and reuse it after changing the task, Harbor refuses to start with FileExistsError: ... already has a lock.json that does not match the resolved job lock. That’s a feature, since it stops you from mixing results from two versions of a task in one job. Use a new name, or delete the old job folder on purpose.

no-network fails loudly where it can’t be enforced. A task can set network_mode = "no-network" in task.toml to say the agent has to work without internet. On the Linux box I used for these examples, a task like that was rejected at startup, because the host couldn’t run Harbor’s egress control, the part that actually blocks the container from reaching the internet. That felt like friction at first. The alternative is a benchmark that quietly runs with internet access while claiming it doesn’t, which is much worse.

My checklist

Before I believe a Harbor number, I go through this list:

1. The oracle passes and nop fails on every task.
2. test.sh writes a 0 on failure instead of crashing. No bare set -e around the test command.
3. At least one lazy baseline, an agent that takes an obvious shortcut, scores 0.
4. Every trial is sorted into a bucket, and the verifier-broken and infra buckets are empty or explained.
5. At least five attempts per task. Report the mean, how often each task is solved every time, and cost per solved trial.

None of this is hard, and none of it needs a model or an API key. It’s the difference between a score you can defend and one you just hope is right.

References

• Companion code for this post: github.com/alireza1989/harbor-benchmarking-example
• Harbor, source code and README: github.com/harbor-framework/harbor
• Harbor documentation: harborframework.com/docs
• Harbor Cookbook, end-to-end examples: github.com/harbor-framework/harbor-cookbook
• Merrill, Shaw, Carlini et al., Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces, arXiv:2601.11868 (2026): arxiv.org/abs/2601.11868
• Terminal-Bench leaderboard and tasks: tbench.ai
• OpenAI, Introducing SWE-bench Verified (August 2024): openai.com/index/introducing-swe-bench-verified
• OpenAI, Why we no longer evaluate SWE-bench Verified (February 2026): openai.com/index/why-we-no-longer-evaluate-swe-bench-verified
• SWE-bench GitHub issue #465, Repo State Loopholes During Agentic Evaluation (September 2025): github.com/SWE-bench/SWE-bench/issues/465
• Chen et al., Evaluating Large Language Models Trained on Code (source of the unbiased pass@k estimator), arXiv:2107.03374 (2021): arxiv.org/abs/2107.03374
• Harbor source referenced above: metrics/base.py (missing rewards count as 0), models/job/config.py (retry defaults), utils/pass_at_k.py (pass@k)