Dashboard

Curator publishes a live dataset analytics databoard:

https://swe-databoard.pages.dev/

Open the Databoard

Difficulty and tag methodology

Difficulty scoring and tagging follow the harbor scripts/task_analysis method, applied uniformly to every dataset.

Difficulty score

A weighted, log-scaled model over five dimensions of the solution patch, tests, and instruction. Each dimension is scaled to 1.0–5.0, combined by weight, then mapped to a final 1.0–10.0 score:

The score is bucketed into a label:

Scoring is static (no LLM) so it is deterministic and reproducible.

Tags

An LLM assigns exactly four tags per task, in order:

[language, area, topic, bug_class]

• language — primary programming language (python, go, typescript, …).
• area — one of backend, frontend, fullstack, cli, library, framework.
• topic — framework/library or focused technical topic (fastapi, react, async, …).
• bug_class — a domain-independent label for the defect mechanism (missing-fallback, incomplete-validation, wrong-default, …), describing the logical failure mode rather than the affected feature.

The tagger disables the model's thinking mode so it returns a strict JSON object, validates the 4-tuple (including the closed area vocabulary), and retries on malformed output. Runs are resumable and idempotent.

Very large or awkward tasks can make the endpoint hang on a full-size prompt. The tagger handles these with progressive truncation: if a request fails, it retries with a shorter patch/instruction/test excerpt (and fewer changed files), stepping down through several levels until the request completes. This guarantees every task receives tags without discarding context for normal-size tasks.

Regenerating the databoard

From blocks/curator/dashboard/:

# 1. Difficulty + tags for every dataset (resumable, 64-way concurrent).
#    The canonical tagger lives in the swegen submodule; --datasets-dir
#    points it at this dashboard's datasets/ directory.
python3 ../repos/swegen/tools/tag_task_metadata.py \
  --datasets-dir datasets --dataset all --jobs 64 --retries 3

# 2. Render the multi-dataset HTML
python3 progress_monitor_multi.py --output-html site/index.html

# 3. Deploy
cd site && CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ACCOUNT_ID=... \
  npx wrangler@latest pages deploy . --project-name=swe-databoard --branch=main

See blocks/curator/dashboard/README.md for dataset export and endpoint configuration.

Docs versus databoard

There are two Cloudflare Pages projects:

Use the docs when onboarding or reviewing the pipeline. Use the databoard to compare dataset difficulty and tag composition.