Benchmarks · no Elo

TerminalBench

The difference between harnesses: same model, same 89 Terminal-Bench 2.1 tasks, one sweep each through the official Harbor adapter. Only the harness changes.

scoring: pass rate · terminal-bench/

What differs: the system prompt and tools (first-call context: TODO for AI 1.1–1.4k tokens, Claude Code and Codex ~15.4k) and refusal handling.How we measuredRun the same agent
HarnessModel (level)SolvedCost (list price)
TODO for AI v1.0.25, bash + tfa-catClaude Opus 5.5 (medium)93.3%$33.88
TODO for AI v1.0.25, bash + tfa-catClaude Opus 5.5 (high)92.1%$51.46
TODO for AI v1.0.11, bash + tfa-catClaude Opus 5.5 (high)91.0%$50.82
TODO for AI, bash + tfa-catGPT-6.1 Sol (high)85.4%$8.63
TODO for AI v1.0.4, read + bashClaude Opus 5.5 (high)89.9%$53.90
Claude Code 2.1.289Claude Opus 5.5 (high)87.6%≈ $45.68
Codex 0.160.1Claude Opus 5.5 (high)84.3%≈ $38.66
TODO for AI v0.9, old promptClaude Opus 5.5 (high)79.8%$31.76
TODO for AI v0.9.27GPT-5.6 Sol (xhigh)79.8%$46.87 promo / $89.60
TODO for AI v0.9.27 + Claude Opus 5 review sub-agentGPT-5.6 Sol (xhigh)82.0%$134.51 promo / $188.33

Cost = tokens × Anthropic list price (Opus 5.5: $5 / $25 per M in/out, $0.5 cache read, $6.25 cache write); ≈ marks totals from the harness's own token report, which misses 2–3 tasks that never produced a trajectory and, for Codex, cache writes. Codex has no refusal fallback (Claude Code and TODO for AI retry on Opus 4.8 when Opus 5.5 refuses on safety grounds), so its 4 refused tasks were rerun on Opus 4.8 by hand and that result counted (3 pass; 72/89 on pure 5.5). The 4.8 rerun is in its cost. GPT Sol rows: OpenAI list price ($2 / $10 per M, $0.10 cache read), no refusal fallback; the 81/89 and 82/89 Sol rows are best-of-attempts composites (the 13 tasks GPT-6.1 Sol high failed got 1–4 extra tries at high/xhigh, 6.2 was not re-run on the 76 that 6.1 already passed), not a clean single sweep. Harness check on the 40 hardest tasks, GPT-6.2 Sol xhigh: TODO for AI 27/40 ($12.72) vs Codex CLI 26/38 ($12.10; it could not install itself on 2 old-Debian tasks), a tie on the 38 both ran. Not a tbench.ai leaderboard submission and not scored by its rules. Methodology.