Benchmarks · no Elo

SubagentBench

Best model for each sub-agent task (explore, webfetch, compaction), on real prompts from our todos.

scoring: judge scores · todo-bench/

Explore sub-agent

r7 · 2026-10-08 · 2 runs each

10 real code-search questions our agents asked (the ones where models differed most), 2 runs each.

ModelScoreHalluc. / runTimeCost (list price, per run)
Claude Haiku 5.56.651.7751s$0.011
GPT-6 Sol6.221.932m 37s$0.35
Claude Opus 5.56.132.732m 20s$0.71
GPT-6 Luna5.602.081m 35s$0.014
Claude Haiku 4.52.775.001m 54s$0.22

Webfetch extraction

r2 · 2026-10-08

177 real url + question pairs from our todos; every model reads the same page snapshot.

ModelScoreHalluc. / runTimeCost (list price, all 177)
GPT-6 Sol9.080.066s$3.67
Claude Haiku 5.58.570.455s$0.55
Claude Opus 5.58.530.538s$12.91
GPT-6 Luna8.410.215s$0.18
Claude Haiku 4.56.671.044s$2.15

Compaction summary

r4 · 2026-10-08

20 real long conversations. The question: can a fresh agent continue from the summary alone?

ModelScoreHalluc. / runTimeCost (list price)
GPT-6 Sol8.950.004m 27s—
Claude Sonnet 5.58.530.7052s—
GPT-6 Luna8.150.721m 46s—
Claude Haiku 5.56.882.3037s—
Claude Haiku 4.54.105.0343s—

Scores 1–10 from two blinded judges (Claude Opus 5.5 and GPT-6 Sol), averaged; on close calls they can disagree (explore, webfetch). Time is the median per call. Cost = tokens × OpenRouter list price (uncached input for webfetch; explore counts its real in-loop cache reads). Scores compare within one round only.