Getting Started

This page walks through preparing the evaluator block and running a benchmark end to end. evaluator wraps a pinned Harbor checkout, so most steps are about getting that runtime in place — there is no task staging step, because the benchmark is resolved from Harbor's registry.

Recommended: operate via the Claude plugin

The block ships a Claude Code plugin (evaluator-plugin) that drives the whole lifecycle through slash commands. This is the recommended way to run evaluator — each command performs the preflight, confirmation, and bookkeeping steps for you. Launch Claude from inside blocks/evaluator/ so the block-local plugin loads, then:

/evaluator:setup          # update repo, build envs, fill config
/evaluator:check          # dryrun preflight: config, repo, envs, LLM endpoint, registry
/evaluator:run            # LiteLLM proxy + Harbor benchmark execution + analysis
/evaluator:dashboard      # browse jobs, reports, and trajectories

The knobs these commands read live in config.yaml — benchmark selection under runtime_info.input.task_source, Harbor job settings under runtime_info.input.harbor_job, and the LLM endpoint under runtime_info.input.llm_api. Edit config.yaml, not the scripts.

The rest of this page documents the underlying commands the plugin runs, for manual operation or debugging.

Prerequisites

• Docker — Harbor rolls each task out inside a container, so a working Docker daemon is required on the run host (evaluator is CPU-only, but every task runs in a container).
• uv — used to build the Harbor environment and run analysis; also required by start.sh (uv run harbor …).
• Python 3.13 — required by the LiteLLM venv.
• PyYAML in system python3 — every scripts/*.sh uses inline python3 - config readers. Run pip install pyyaml if you see ERROR: PyYAML is required.
• An OpenAI- or Anthropic-compatible LLM endpoint — the model the agent will call, fronted by a per-job LiteLLM proxy. This can be a remote API or a local checkpoint served with vLLM (see Local Model).

1. Update the managed repo

evaluator depends on one local-only repository, harbor, pinned to a specific commit in config.yaml. Clone or update it to the pinned ref:

bash scripts/update_repos.sh

The script clones/fetches, checks out the pinned commit detached, initializes submodules, and sets the worktree read-only (readonly: true). It refuses to update a worktree with local modifications. The block reads registry.json from this checkout.

2. Build the environments

evaluator has no setup_*_env.sh helper scripts (unlike tracer). The two environments are built with explicit uv commands — scripts/dryrun.sh also prints these when an env is missing. There is no swe_data_process env, because evaluator does not convert trajectories.

# Harbor uv env — build from inside the read-only repo, to a path OUTSIDE it
( cd repos/harbor && UV_PROJECT_ENVIRONMENT="$PWD/../../artifacts/env/harbor-uv" uv sync --all-extras )

# LiteLLM venv — Python 3.13, pinned litellm
uv venv artifacts/env/litellm-venv --python 3.13
uv pip install --python artifacts/env/litellm-venv/bin/python 'litellm[proxy]==1.83.14'

The Harbor env path must be outside repos/harbor while the repo is read-only — artifacts/env/harbor-uv already satisfies this.

3. Pick a benchmark

evaluator is registry-driven and stages no tasks locally. Select a benchmark by editing config.yaml → runtime_info.input.task_source:

task_source:
  provider: harbor_registry
  dataset_name: swebench-verified   # or any supported benchmark
  version: "1.0"
  registry_path: repos/harbor/registry.json
  no_hack: false                    # true = swebench-verified with agent egress allowlist

See Select Benchmark for the full table and the no_hack option. For a fast first run, pick a -100 subset (e.g. swebench-verified-100) or set harbor_job.n_tasks to a small integer.

4. Validate the config

Run the static dry-run and then the live completion probe. Together they validate the config, Harbor repo state, both environments, registry entry, and the real upstream completion path:

bash scripts/dryrun.sh
bash scripts/probe_llm_completion.sh

Fix anything it reports before launching a job.

5. Launch a job

Only after both gates pass, launch the evaluator. start.sh repeats them, generates a per-job LiteLLM config, starts the proxy on the configured port, builds the Harbor command (--dataset <name>@<version> --registry-path repos/harbor/registry.json, plus any --exclude-task-name flags), runs the job, and then runs post-eval job analysis:

bash scripts/start.sh

6. Inspect results

Each rollout writes a trajectory and per-task evaluation artifacts to:

artifacts/jobs/<job>/<task>/agent/litellm-trajectory.jsonl
artifacts/jobs/<job>/<task>/verifier/          # per-task verdict, scoring, test logs

Post-eval analysis lands under artifacts/jobs/<job>/analysis/. For a visual view, open the dashboard.