Scoring
Not every successful rollout makes useful training data. The pinned
swe_data_process package provides one deterministic scorer that runs during
conversion and two optional LLM-as-judge scorers. Their results coexist in the
IM record's _score metadata.
TQS V2 rule scoring
rule_score.py runs automatically after Raw → IM conversion. TQS V2 uses a
fail-soft weighted aggregate: a component with insufficient evidence is
null and is omitted from both the numerator and denominator.
composite_score =
Σ(weight × transformed_component) / Σ(active_weights)Five components have non-zero weights:
| Component | Weight | Meaning |
|---|---|---|
SUB | 0.33 | Submission completeness, including how the trajectory ends and late-run quality |
STP | 0.27 | Step efficiency based on assistant-turn count |
TVR | 0.23 | Test-writing, test execution, and late successful verification |
FEC | 0.10 | File-edit concentration; aggregated as FEC^5 |
DPI | 0.07 | Dirty-pattern penalty for truncation, no successful write, loops, and repeated errors; aggregated as DPI^3 |
The scorer also emits diagnostic-only components with zero aggregate weight:
OEC (observation entropy collapse), IAC (intent/action consistency), PED
(perception/edit drift), PSN (path stability), TTE (tool-transition
entropy), and SCP (successful-change position).
TQS V2 supports Claude Code, OpenCode, OpenHands/OpenHands SDK, and Terminus-2. It normalizes renamed tools by their parameter schema before detection and limits command-output error matching to execution tools, avoiding false positives from source code returned by read/edit tools.
Main-agent records receive _score; subagent records are retained with
_score: null. Scores are copied into LF metadata and summarized in
lf.stats.json.
Fixed-checklist LLM scoring
llm_score.py is optional. An OpenAI-compatible judge rates a fixed checklist
of 15 checks on a 1–5 scale:
| Category | Focus |
|---|---|
| F | Problem understanding |
| G | Solution quality |
| H | Reasoning quality |
| I | Verification rigor |
| J | Efficiency |
Each category and the overall llm_composite_score are normalized to [0, 1]
and stored with an llm_ prefix. This scorer makes one judge call per
main-agent record and supports dry-run cost estimation.
Dynamic-checklist LLM scoring
llm_checklist_score.py is an optional OctoBench-aligned alternative. It first
generates 15–35 binary checks from the user request, system prompt, tool
schema, and repository policies, then makes a second call to judge the
trajectory.
It reports:
llm_checklist_isr—1.0only when every required check passes.llm_checklist_csr— the fraction of individual checks that pass.- Category-level CSR values and the generated checklist/judgement evidence.
The fixed and dynamic LLM scores are independent of TQS V2. Run either only when its semantic signal justifies the additional endpoint cost.
Using scores
Use composite_score, llm_composite_score, or checklist ISR/CSR to inspect,
filter, or curriculum-order data. Keep main and subagent records bundled when
filtering an instance; swe_data_process provides bundled score filtering for
that purpose.
The complete rubrics and CLI options live in the pinned checkout:
repos/swe_data_process/docs/rule_score_details.md
repos/swe_data_process/docs/llm_score_details.md
repos/swe_data_process/docs/llm_checklist_score_details.md