Select Benchmark

evaluator is registry-driven: it does not stage tasks locally. Switch benchmark by editing fields in config.yaml → runtime_info.input.task_source:

task_source:
  provider: harbor_registry
  dataset_name: swebench-verified
  version: "1.0"
  registry_path: repos/harbor/registry.json
  no_hack: false

Harbor resolves (dataset_name, version) against repos/harbor/registry.json and fetches the underlying task data automatically. No artifacts/tasks/ staging is required.

Nohack (agent egress control)

Set task_source.no_hack: true to run the hardened SWE-Bench Verified variant that limits agent network egress to the per-job LiteLLM host (verifier stays public so the SWE-Bench parser can still resolve PyPI deps).

When enabled, start.sh:

1. Keeps dataset_name: swebench-verified as the source in config
2. Runs scripts/prepare_nohack.sh to generate (if missing) the local dataset under repos/harbor/datasets/swebench-verified-nohack and registry at repos/harbor/scripts/git_ignore/hack_control/registry.swebench_verified_nohack.json
3. Launches Harbor against swebench-verified-nohack@1.0 with --agent-extra-allowed-host <LiteLLM host>

Currently only dataset_name: swebench-verified is supported. The -100 subset is rejected with no_hack — the remap would silently expand it to the full 500-task set; cap a smoke run with harbor_job.n_tasks instead. n_tasks also limits nohack registry generation to a separate registry….limitN.json so a short probe does not replace the full nohack registry (stale limitN variants are removed by scripts/clean.sh). Refresh a stale local nohack tree with NOHACK_REFRESH=1 before start.sh.

See Harbor's docs_dev/network-policy-nohack.md for the phase-level network policy details.

Curated benchmarks

These are validated to run end-to-end under the agents eval configures (e.g. custom-claude-code):

Smoke runs

For a fast run, either pick a -100 subset from the table, or cap the full benchmark with harbor_job.n_tasks:

harbor_job:
  n_tasks: 16   # null = full benchmark; an int caps the run

Other registry entries

The table above is the curated set. The full registry.json contains ~80 more entries (e.g. terminal-bench-pro, swesmith, swtbench-verified, plus non-code benchmarks like gpqa-diamond, aime, gaia, lawbench, …). The scripts accept any of them as dataset_name without code changes, but two caveats apply:

• Agent compatibility is your problem. Math/MCQ/QA benchmarks (aime, gpqa-diamond, simpleqa, …) and benchmarks that ship their own agent runtime (gaia, mlgym-bench, …) generally do not work as-is with the coding agents configured here. Verify against the adapter README under Harbor's adapters/<name>/ before adding one to a production run profile.

• List the full set with:

  python3 -c 'import json; [print(e["name"]+"@"+e["version"]) for e in json.load(open("repos/harbor/registry.json"))]'