LegoFlow

BlockstracerReference

Inputs & Outputs

Tracer is configured through config.yaml. The file has exactly meta_info and runtime_info as top-level sections. Treat it as the run profile and dependency contract, not as live status.

Managed repositories and environments

meta_info.repositories pins:

  • harbor — the rollout runtime.
  • swe_data_process — Raw → IM → LF conversion and quality scoring.

meta_info.environment places writable environments outside those read-only checkouts:

KeyCurrent path/purpose
harbor_uvartifacts/env/harbor-uv
litellm_uvartifacts/env/litellm-venv, Python 3.13 + LiteLLM 1.83.14
swe_data_process_uvartifacts/env/swe-data-process-uv
swe_data_process_extrasOptional package extras; currently includes llm

Runtime inputs

InputPurpose
llm_apiUpstream key, base URL, model, protocols, and token pricing for the per-job proxy
litellm_proxyHarbor template, local port, and local master key
task_sourcelocal curator output or a huggingface dataset
harbor_jobJobs directory, task cap, concurrency, retries, and timeout multiplier
agentRollout agent identity, version, image/runtime mount, turns, and sampling
sft_conversionOptional conversion, scaffold, tokenizer, filters, and output directory
env_extra.HARBOR_EXCLUDE_TASKSExclusion sources: excluded_tasks.txt (human, tracked) + artifacts/processed_tasks.yaml (run history); literal ids also accepted

Verifying the endpoint

llm_api.api_base_url must be the OpenAI-compatible base including /v1, and llm_api.model carries the openai/ prefix LiteLLM expects. Check both with a real completion rather than a catalog listing:

bash scripts/probe_llm_endpoint.sh          # 0 PASS, 1 FAIL, 77 SKIP

It requires a non-empty text reply — a gateway can answer GET /models from local config while every completion fails, and a reasoning model can return HTTP 200 with empty content. On failure it retries the neighbouring URL/model shapes (missing /v1, stray provider prefix) and prints the exact edit. CI runs it as tests/cases/04_llm_endpoint.sh.

The current profile is summarized below. API secrets remain placeholders:

llm_api:
  api_key: human
  api_base_url: human
  model: "openai/Qwen3.6-35B-A3B"
  protocols: [openai_compatible, anthropic_compatible]
  served_via: per_job_litellm_proxy

litellm_proxy:
  config_template: scripts/serve_llm/litellm_config.example.yaml
  port: 4003
  master_key: dummy-key-cf

task_source:
  provider: local
  dataset_name: ../curator/artifacts/merged_swe_tasks
  split: train

harbor_job:
  jobs_dir: artifacts/jobs
  n_concurrent: 8
  n_tasks: 40
  max_retries: 2
  timeout_multiplier: 5

agent:
  name: custom-claude-code
  version: 2.1.118
  runtime_image: docker.io/jierun/c-cc-2.1.118:v0.1
  runtime_host_path: artifacts/agent-runtime/claude-code
  max_turns: 80
  temperature: 0.7

sft_conversion:
  enabled: false
  scaffold: auto
  tokenizer_name: Qwen/Qwen3.5-35B-A3B-Base
  out_dir: artifacts/sft_data
  max_instances: null
  reasoning_check_mode: adaptive
  reasoning_content_ratio_threshold: 0.2

Always read the checked-in config before a run; these documented defaults are an orientation aid, not a replacement for validation.

Dependency wiring

With task_source.provider: local, tracer consumes:

curator.output.merged_tasks_dir
  → tracer.input.task_source.dataset_name

With trainer.input.source.type: harbor_job, tracer publishes:

tracer.output.raw_trajectories_dir
  → trainer.input.source.job_dir

Both edges are declared by their producer and consumer. The shared validator reports missing, mismatched, unresolved, or path-incompatible edges.

Outputs

OutputPathFormat and consumer
raw_trajectories_dirartifacts/jobs/Harbor jobs with artifacts/jobs/<job>/<task>/agent/litellm-trajectory.jsonl; wired to trainer
sft_data_dirartifacts/sft_data/Optional artifacts/sft_data/<job>/lf.json in LLaMA-Factory format; declared for direct consumption

Job results, conversion sidecars, and archives are described in Results & Artifacts.

On this page