Getting Started
Prerequisites
Before starting, make sure you have the pieces that the pipeline cannot create for you:
- Claude Code, which is the recommended way to operate LegoFlow.
- LLM endpoints for curator, tracer, and evaluator jobs.
- GitHub token(s), used by curator when collecting pull requests.
- A GPU server if you plan to run training or evaluate self-hosted models.
- Optional Docker credentials, useful when large runs pull many task images.
- Optional Cloudflare credentials, only needed if you want to publish dashboards.
Claude Code Plugin Installation
The easiest way to use LegoFlow is through plugin skills. Think of these as guided commands that know the project layout, check the right files, and ask for confirmation before expensive jobs start.
At the repository root, load the root plugin and all block plugins together:
claude \
--plugin-dir .claude/plugins/root-plugin \
--plugin-dir blocks/curator/.claude/plugins/curator-plugin \
--plugin-dir blocks/tracer/.claude/plugins/tracer-plugin \
--plugin-dir blocks/trainer/.claude/plugins/trainer-plugin \
--plugin-dir blocks/evaluator/.claude/plugins/evaluator-pluginAfter installing plugin files, run /reload-plugins in Claude Code. Each block inherit similar skills and usages, e.g., /block:setup, /block:check, and /block:run. Users can easily trigger these blogs with these pre-defined skills.
Project Setup
Once the plugins are loaded, start with setup:
/root:setupThis is safe to run more than once. It checks shared tooling, verifies the root
config.yaml, and can walk into each child block's setup flow. It does not
launch the pipeline; it only gets the workspace ready for a real check.
The setup flow keeps block-specific decisions close to the block that uses them:
- curator prepares task-curation dependencies and GitHub/LLM input channels;
- tracer prepares rollout dependencies, task sources, agent runtimes, and LLM endpoints;
- trainer prepares data conversion and training dependencies, including GPU-side requirements;
- evaluator prepares benchmark, model, and analysis dependencies.
For block-specific details, open the corresponding page under Blocks.
Running Individual Blocks (Recommended)
If you are new to LegoFlow, start by running one block at a time. This gives you a much clearer sense of what each stage consumes, produces, and checks before you ask the root block to run the whole pipeline.
Each block's own Getting Started page explains how to fill its config.yaml,
which credentials it expects, and what outputs it writes. Keep those details in
the block pages so this root guide can stay focused on the overall workflow.
Step 1: Task Curation
Curator turns source signals such as GitHub PRs, issues, and online forums into verified SWE tasks.
| Skill | Purpose |
|---|---|
/curator:setup | Prepare curator dependencies, repositories, GitHub tokens, and LLM inputs. |
/curator:check | Validate curator config, repo pins, tokens, Docker, and LLM endpoint readiness. |
/curator:collect-prs | Collect candidate PRs for downstream task generation. |
/curator:create-tasks | Generate and validate SWE tasks from the collected candidates. |
/curator:dashboard | Inspect curated task statistics, filters, and dataset quality signals. |
Use the curator page for task-source choices, GitHub token setup, filtering rules, and output formats: Curator.
Step 2: Trajectory Rollout
Tracer takes curated tasks and rolls out coding agents in isolated environments. This is where task instances become trajectories.
| Skill | Purpose |
|---|---|
/tracer:setup | Prepare rollout repositories, environments, task sources, agent runtimes, and LLM inputs. |
/tracer:check | Validate tracer config, task source, proxy port, runtime image, repos, and endpoint readiness. |
/tracer:run | Launch trajectory rollout jobs for the configured tasks and agent scaffold. |
/tracer:dashboard | Monitor rollout jobs, trajectory status, archives, and SFT conversion progress. |
Use the tracer page for task-source configuration, agent presets, concurrency, job outputs, and trajectory conversion: Tracer.
Step 3: Training
Trainer converts trajectories into training-ready data and launches the model training workflow.
| Skill | Purpose |
|---|---|
/trainer:setup | Prepare training repositories, Python environment, data conversion stack, and GPU-side dependencies. |
/trainer:check | Validate trajectory source, model path, DeepSpeed config, GPU visibility, and training environment. |
/trainer:run | Convert trajectories and launch the configured training job. |
/trainer:dashboard | Inspect training status, metrics, artifacts, and run comparisons. |
Use the trainer page for dataset inputs, model paths, GPU settings, and training artifacts: Trainer.
Step 4: Evaluation
Evaluator runs benchmarks and analysis over configured models or checkpoints.
| Skill | Purpose |
|---|---|
/evaluator:setup | Prepare benchmark assets, Harbor environment, LiteLLM proxy, model endpoint, and agent runtime. |
/evaluator:check | Validate benchmark registry, runtime image, agent bind mount, proxy port, and completion readiness. |
/evaluator:run | Run evaluation jobs for the configured model or checkpoint. |
/evaluator:dashboard | Inspect evaluation jobs, scores, trajectories, and analysis reports. |
Use the evaluator page for benchmark selection, model endpoints, local vLLM usage, result files, and analysis dashboards: Evaluator.
Running Cascaded Blocks
Use the cascaded mode after you have run through the individual blocks at least once. The root block is useful when the handoffs are already understood and you want LegoFlow to coordinate the full workflow for you.
Step 1: Check the Tree
Before a cascaded run, ask the root block to check every stage:
/root:checkThis is read-only. It validates configs, dependency handoffs, pinned repos, runtime resources, block-specific dry runs, and live LLM endpoint availability. Fix every required failure before continuing. Warnings such as optional Cloudflare credentials or Docker login do not block execution, but they are worth reading before a large run.
Step 2: Launch the Pipeline
When the check says the tree is safe to run, start the cascaded workflow:
/root:run start the data pipelineThe root skill repeats preflight, shows the run configuration, and waits for your confirmation before starting expensive work. In full-pipeline mode, it dispatches blocks in dependency order:
curator -> tracer -> trainer -> evaluatorEach completed leaf-block run writes an archive:
artifacts/archives/run_NNN/
├── metadata.yaml
├── config.yaml
└── scripts/and appends an entry to artifacts/index.yaml. Job-specific results remain
under the block's declared output directories, such as
blocks/tracer/artifacts/jobs/.
Results Inspection
LegoFlow build dashboards for each concrete blocks. [introduce the benefits of using dashboard and how it can be used] Check [xxx] for more details.