capability · T1 · reasoning
Transitive height failed
cap_reas_002
Score
0%
Turns
1
1 useful · 0 useless
Interventions
0
harness hints (HHI)
Tool calls
0/0
matched expected trajectory
Errors
0/0
recovered / total
Latency
268ms
Tokens
55
0 reasoning
Cost
$0.00001
reasoning_error — CoT contradicts answer / bad chain · expected "dana", got "Carol"
Trajectory timeline
Turn-by-turn replay · useless turns annotated with the classifier reason
- #0modelusefulresponse
Prompt / output inspector
instruction
Alice is taller than Bob. Bob is taller than Carol. Dana is shorter than Carol. Who is the shortest? Answer with the name only.
final output
Carol
expected
Dana
Spans
1 recorded · tokens & cost per span
- generate268ms53 in · 0 cached · 2 out · $0.00001 · ttft 249ms
Events
- task_started{"trial":0,"taskId":"cap_reas_002","scaffold":null}
- task_finished{"score":0,"passed":false,"taskId":"cap_reas_002","failureLabel":"reasoning_error"}
Scorer metadata
{
"toolLog": [],
"scaffold": null
}Capabilityexact_matchdifficulty 1/5