Board
Multi-model matrix
Latest finished run per model · cells are heat-coloured by direction-aware quality · click a header to sort. Fairness: same harness, tasks, tools and judge; costs normalized; judge κ reported.
Board as a landscape
Rows = models (sorted as the table) · columns = Omni + five pillar composites · orbit, hover for exact values
Rendering 3D view…
| # | Model | Elo | Omni score ↓ | Capability score | Reliability score | Safety score | Agency score | Economics score | Battery pass rate | Task success | Hand-Holding Index | OWASP pass rate | ECE | Cost / solved task | TTFT | κ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Swift Budget (simulated) sim:swift-budget | 1510 | 82.0 | 67.9 | 72.2 | 90.0 | 94.2 | 85.6 | 67.9% | 100.0% | 0.25 | 90.0% | 0.333 | $0.00008 | 310ms | 0.83 |
| 2 | Atlas Frontier (simulated) sim:atlas-frontier | 1570 | 79.5 | 96.4 | 91.7 | 100.0 | 77.5 | 31.8 | 96.4% | 100.0% | 0.08 | 100.0% | 0.080 | $0.0107 | 665ms | 0.83 |
| 3 | Local Llama 8B (simulated) sim:local-llama · local | 1420 | 58.4 | 46.4 | 62.3 | 40.0 | 53.2 | 89.9 | 46.4% | 75.0% | 1.17 | 40.0% | 0.347 | $0.00005 | 907ms | 0.83 |
Pairwise Elo
Task-level win/loss/tie across shared tasks (trial 0), K=32, 4 passes
| Model | Elo | W | L | T |
|---|---|---|---|---|
| sim:atlas-frontier | 1570 | 43 | 2 | 87 |
| sim:swift-budget | 1510 | 23 | 21 | 88 |
| sim:local-llama | 1420 | 8 | 51 | 73 |
sim:local-llama vs sim:swift-budget: 7–22 (37 ties)
sim:local-llama vs sim:atlas-frontier: 1–29 (36 ties)
sim:swift-budget vs sim:atlas-frontier: 1–14 (51 ties)
Efficiency frontier
Capability vs. cost per solved task — Pareto-optimal models highlighted