Core Concepts

tracer has the following core concepts:

Verified task and manifest

A task is a single Harbor task directory: an instruction, a container environment, and tests. Tracer does not author tasks. It consumes either the upstream curator output or a compatible Hugging Face dataset.

For a local source, curator can provide a manifest, verifiable_tasks.txt, containing the task IDs that passed verification. scripts/prepare_tasks.sh filters strictly through it when it is present. The current compatibility behavior for a local source without a manifest is to copy all structurally valid task directories, with a message explaining the fallback. Hugging Face sources are staged from their compatible packaged task directories.

Trajectory

A trajectory is the complete, replayable log of one agent rollout. tracer captures it via the LiteLLM logger at:

artifacts/jobs/<job>/<task>/agent/litellm-trajectory.jsonl

Each line records a model request/response together with token usage and metadata. Trajectories are the block's primary output and the raw material for SFT data.

Rollout / trial

A rollout (Harbor calls it a trial) is one attempt by the agent to complete a task. A rollout produces a reward — typically 1.0 if the task's test script passes and 0.0 otherwise. tracer can retry failed rollouts up to max_retries.

Job

A job is a collection of rollouts across the prepared task set, driven by a single config.yaml. Harbor runs up to n_concurrent rollouts in parallel and writes an aggregate result.json plus per-task trajectories under artifacts/jobs/<job>/.

LiteLLM proxy

tracer starts a per-job LiteLLM proxy in front of your upstream model API. The proxy normalizes the endpoint so the agent can speak OpenAI- or Anthropic-compatible protocols, attaches the trajectory logger, and is torn down when the job ends. Each job gets its own generated config under artifacts/litellm/<job>/.

Agent scaffold

A scaffold is the agent harness that drives the model through a task — for example Claude Code, OpenCode, OpenHands SDK, or Terminus-2. The scaffold determines the raw trajectory shape, which in turn selects the converter used for SFT data. See SFT scaffolds.

Processed-tasks ledger

The processed-tasks ledger (artifacts/processed_tasks.yaml) is the source of truth for task states (pending | running | done | failed | skipped) and for what this block already consumed. It is runtime state: artifacts/ is gitignored, so the ledger does not survive a fresh clone. Tasks a human retired for good live in the git-tracked excluded_tasks.txt instead.

runtime_info.input.env_extra.HARBOR_EXCLUDE_TASKS names both files rather than listing ids, so config.yaml stays one line and cannot drift from them. start.sh resolves it at launch into --exclude-task-name flags. The tracer plugin updates the ledger after inspecting each job; without that bookkeeping, a completed task is not protected on a future run.

SFT data (IM and LF)

tracer converts raw trajectories into training data in two stages:

• IM (intermediate) — PangUML v2 records containing OpenAI-style messages, tools, metadata, and an optional TQS score, one JSONL row per converted trajectory.
• LF (LLaMA-Factory) — ShareGPT-style messages as a JSON array, ready to feed the trainer block.

See SFT Data for the full conversion pipeline.