Loading tasks…
Context
Expected
Structured answer
Exact prompt
Mock check
Free controls: a known correct answer and a non-answer.
Loading results…
UnvalidatedPilotNot a ranking
CLI pilot comparison
Same frozen tasks, one attempt each. PASS/FAIL and end-to-end CLI latency only.
Loading comparison…
Tasks
The frozen india-pilot set. Private prompts are omitted from this preview.
Proposed tasks
Design seeds only — not executable, no results. Excluded from this preview pending release review.
Run
Plan, execute and inspect jobs. Execution is disabled in this static preview — run locally with km ui.
01 · Plan
Tasks and adapters
Each try is a fresh attempt with its own seed. More tries show how consistent a system is and multiply time and cost.
Adapters
Controls validate the evaluator; they never show model capability.
CLIs have no sampling seed; the seed identifies the evaluator run.
02 · Execute
Progress
Select tasks and adapters to start a saved run.
Cancel stops remaining calls; saved attempts survive. Retry creates a new evidence trail.
Saved jobs
Consumer apps
Consumer clients are not automated. Record their answers instead; a CLI run does not evaluate a web-app tier.
03 · Inspect
Results and evidence
Results appear as attempts are saved.
Recorded answers
Score answers you recorded from apps, side by side. Nothing is called.
1
Task
Context
Expected
Exact prompt and scoring rule
Correct only when every expected JSON field matches with no extras. Completion is recorded separately.
2
Configurations
Select 2–8. Plan labels do not verify account access.
Add a configuration
3
Answers
Paste comparison JSON
SAMPLE ANSWERS Synthetic controls only. No app was evaluated.
Unknown timing and cost stay unknown. A subscription fee is not a per-task price.
App observations
Record what a consumer app returned. App, client and plan are part of the evidence.
Copy the exact prompt, try it in your app, then record the response and evidence. This form never signs in or runs the app.
Saved
Local mock checks and app observations. Tracks are never mixed.
Select a record to inspect it.
About
A local-first evaluator: models, CLI harnesses and apps on the same frozen tasks, graded deterministically.
This preview
- Status
- Unvalidated / pilot Not a ranking
- Suite
- india-pilot · 24 tasks · 13 Oct 2026 IST
- Claude Code 2.1.296
- claude-opus-5-5 · 23/24 · median 4.1 s
- Muse Code 1.4.7
- muse-spark-1.3 · 22/24 · median 26.7 s
- Wonder Max
- Unavailable. Muse Spark is not Wonder Max.
- Latency
- Observed end-to-end CLI run latency — not intrinsic model speed.
- Conditions
- Same prompt, high effort, no tools, first attempt, deterministic graders.
Privacy
- PASS/FAIL and latency aggregates only
- No prompts, model answers or raw logs
- No credentials, secrets or local paths
- No backend and no network calls
Run it locally
Live runs, imports and recording are local-only (GitHub).
uv sync
uv run km ui --port 8765