LiteLLM Proxy

tracer never points the agent directly at your upstream model API. Instead, each job starts its own LiteLLM proxy, which normalizes the endpoint, exposes both OpenAI- and Anthropic-compatible protocols, and attaches the trajectory logger.

Why a proxy

• Protocol bridging — different agent scaffolds expect OpenAI or Anthropic message formats. The proxy serves both from one upstream model.
• Trajectory capture — the proxy's logger writes each request/response to litellm-trajectory.jsonl, which is what makes the run reproducible and convertible to SFT data.
• Isolation — each job gets a fresh, per-job config so concurrent or sequential jobs don't share state.

Configuration

The upstream model is declared in config.yaml under runtime_info.input.llm_api:

llm_api:
  api_key: <your-key>
  api_base_url: "https://your-openai-compatible-endpoint/v1"
  model: "openai/Qwen3.6-35B-A3B"
  protocols: [openai_compatible, anthropic_compatible]
  served_via: per_job_litellm_proxy
  input_cost_per_token: 0.0000021
  output_cost_per_token: 0.0000084

The proxy itself is configured under runtime_info.input.litellm_proxy:

litellm_proxy:
  config_template: scripts/serve_llm/litellm_config.example.yaml
  port: 4003
  master_key: dummy-key-cf

The master key is the local proxy credential and may be a functional dummy. The upstream llm_api.api_key is real and must not be committed.

Lifecycle

scripts/start.sh handles the proxy automatically:

1. Renders a per-job config from the template into artifacts/litellm/<job>/litellm_config_tracer.yaml.
2. Starts the proxy on litellm_proxy.port, passing the config to Harbor's LiteLLM serve script via LITELLM_CONFIG.
3. Starts the dedicated LiteLLM 1.83.14 environment and waits for the port to become reachable.
4. Runs the Harbor job against the proxy.
5. Stops the proxy from the EXIT trap and archives the run, including failures and interruptions.