Update, 30 September: Opus 5.5
We ran the same 89 tasks with Claude Opus 5.5 on the same harness as the Sol review-off run: bash, read and web fetch, an isolated single-machine bridge, no review sub-agent. The headline is 95.1% (77/81).
Why 81, not 89. Eight tasks never ran. Anthropic’s safety classifier flags them at the start (three biology tasks: dna-assembly, dna-insert, protein-assembly; five security tasks: break-filter-js-from-html, filter-js-from-html, crack-7z-hash, password-recovery, vulnerable-secret), the todo stops after one or two calls, and Harbor records an ApiError. The model did not attempt them, so in this score they count as neither a pass nor a fail and are left out of the denominator. Retries mostly refuse again (in a retry of four of them, three were refused), so we do not retry them. Counting them as failures gives 86.5% (77/89); both numbers are in the results sheet.
| Opus 5.5 | GPT-5.6 Sol, review off | |
|---|---|---|
| Same 81 tasks (refused excluded) | 95.1% (77/81) | 82.7% (67/81) |
| Wilson 95% interval, 81 tasks | [88.0%, 98.1%] | [73.1%, 89.4%] |
| All 89 tasks | 86.5% (77/89) | 82.0% (73/89) |
| Reasoning | xhigh sweep; high on 6 reruns | xhigh |
| Dates | 26–30 September 2026 | 2–4 September 2026 |
The Sol column uses Sol’s final review-off result (73/89 after a later rerun fixed a token-expiry bug; the 79.8% below is the earlier 71/89 snapshot). Sol does attempt the eight tasks and passed six of them, so the exclusion favours Opus: it lifts Opus by 8.5 points and Sol by 0.7. Counting all 89 tasks, Opus still leads, 86.5% to 82.0%, but by less. The intervals overlap, so on 81 tasks the gap is suggestive, not proven.
Cost. $76.75 at list price for the sweep plus the timeout and polyglot reruns; the two 30 September qemu/video reruns are not yet priced, so treat it as a lower bound. It is not directly comparable with Sol’s $61.87, which counts the last attempt per task only.
Reruns, and one that favours us. Nine tasks are scored on a later run, and the last run counts whether it passed or failed:
- Harness or verifier fixes, 4 tasks.
polyglot-c-pystalled ~300 s per attempt on an interactivetzdataprompt until the harness setDEBIAN_FRONTEND=noninteractive.qemu-startupandqemu-alpine-sshwere solved by the agent, but the task’s own verifier (ondebian:bullseye) could not installcurl/sshpassbecause the end-of-life security mirror returns 404, so every run scored 0; the harness now drops that mirror before the verifier runs.extract-moves-from-videowas rerun after a CPU-limit fix and still failed. - Timeouts, 6 tasks, rerun at
highreasoning:adaptive-rejection-sampler,cobol-modernization,extract-moves-from-video,make-doom-for-mips,query-optimize,schemelike-metacircular-eval. These were agent timeouts, not infrastructure damage, so rerunning them is a second attempt. Two flipped to pass (adaptive-rejection-sampler,cobol-modernization), four still failed. Without those two the score is 92.6% (75/81).
Each square is one attempted task; the 8 refused tasks are not shown. Hatched squares were scored on a later run. Hover for the task ID.
All task IDs and outcomes
bn-fit-modify— passedbuild-cython-ext— passedbuild-pmars— passedbuild-pov-ray— passedcaffe-cifar-10— passedcancel-async-tasks— passedchess-best-move— passedcircuit-fibsqrt— passedcode-from-image— passedcompile-compcert— passedconfigure-git-webserver— passedconstraints-scheduling— passedcount-dataset-tokens— passedcustom-memory-heap-crash— passeddb-wal-recovery— passeddistribution-search— passedextract-elf— passedfeal-differential-cryptanalysis— passedfeal-linear-cryptanalysis— passedfinancial-document-processor— passedfix-code-vulnerability— passedfix-git— passedfix-ocaml-gc— passedgcode-to-text— passedgit-leak-recovery— passedgit-multibranch— passedgpt2-codegolf— passedheadless-terminal— passedhf-model-inference— passedinstall-windows-3.11— passedkv-store-grpc— passedlarge-scale-text-editing— passedlargest-eigenval— passedllm-inference-batching-scheduler— passedlog-summary-date-ranges— passedmailman— passedmake-mips-interpreter— passedmcmc-sampling-stan— passedmerge-diff-arc-agi-task— passedmodel-extraction-relu-logits— passedmodernize-scientific-stack— passedmteb-leaderboard— passedmteb-retrieve— passedmulti-source-data-merger— passednginx-request-logging— passedopenssl-selfsigned-cert— passedoverfull-hbox— passedpath-tracing— passedpath-tracing-reverse— passedpolyglot-rust-c— passedportfolio-optimization— passedprove-plus-comm— passedpypi-server— passedpytorch-model-cli— passedpytorch-model-recovery— passedraman-fitting— passedregex-chess— passedregex-log— passedreshard-c4-data— passedrstan-to-pystan— passedsam-cell-seg— passedsanitize-git-repo— passedsparql-university— passedsqlite-db-truncate— passedsqlite-with-gcov— passedtorch-pipeline-parallelism— passedtorch-tensor-parallelism— passedtrain-fasttext— passedtune-mjcf— passedvideo-processing— passedwinning-avg-corewars— passedwrite-compressor— passedadaptive-rejection-sampler— passed, replacement runcobol-modernization— passed, replacement runpolyglot-c-py— passed, replacement runqemu-alpine-ssh— passed, replacement runqemu-startup— passed, replacement runextract-moves-from-video— failed, replacement runmake-doom-for-mips— failed, replacement runquery-optimize— failed, replacement runschemelike-metacircular-eval— failed, replacement run
The four failures: schemelike-metacircular-eval (timeout, long thinking turns), make-doom-for-mips (timeout), extract-moves-from-video (timeout: OCR of 951 frames on one CPU) and query-optimize (wrong answer).
Comparison. The 78.9% (Claude Code, Opus 4.8) and 78.4% (Codex, GPT-5.6 Terra) on our site are tbench.ai leaderboard entries over all 89 tasks, several trials each; the same non-equivalences as below apply, and refusals there count as failures. Treat the gap as indicative.
The rest of this post documents the earlier GPT-5.6 Sol runs.
Earlier runs: GPT-5.6 Sol
Before Opus 5.5, we ran Terminal-Bench 2.1 twice on the same 89 tasks with GPT-5.6 Sol. The headline then was the single-model run: 79.8% (71/89), GPT-5.6 Sol at xhigh, with the review tool denied. Web fetch still delegates page reading to Claude Haiku 4.5 ($4.17 of the $46.87); nothing in that run reviews or checks the main model’s work.
The earlier run, 25–26 August, had a review sub-agent on Claude Opus 5 switched on and scored 82.0% (73/89). The difference is two tasks for 2.9× the cost, and the tasks review gained were the “almost” ones: 1 of 2 chess moves, 3 of 4 tests. Most of this post documents that earlier run, because it is the one with a per-task outcome table; the review-off run uses the same harness, the same scoring rule and the same pinned repository, and its results sheet lists every change and every failure.
| Review off (headline) | Review on | |
|---|---|---|
| Pass rate | 79.8% (71/89) | 82.0% (73/89) |
| Wilson 95% interval | [70.3%, 86.8%] | [72.8%, 88.6%] |
| Models | GPT-5.6 Sol; Haiku 4.5 for web fetch | + Claude Opus 5 review |
| Cost, promo / list | $46.87 / ~$89.6 | $134.51 / $188.33 |
| Dates | 2–3 September 2026 | 25–26 August 2026 |
The sections below document the review-on run of Terminal-Bench 2.1 on 25–26 August 2026. This post documents how that number was produced so it can be read correctly: it is a system result under a stated rerun policy, not an official leaderboard submission and not a single-model score.
Every figure below is taken from the pinned results sheet in the public benchmark repository. Links pin a commit, so later edits cannot silently change what this article describes.
The system under test
Terminal-Bench gives an agent a task in an isolated container — compile a project, recover a database, configure a service, fix code — and a verifier checks the resulting state. The agent’s own claim of success is irrelevant; only the verifier’s outcome counts.
- 89 task IDs
- isolated Docker env
- sends instruction
- edge executes in container
- served model verified in message metadata
- review → Claude Opus 5
- explore, webfetch → Haiku 4.5
- one final outcome per task
Two details of this configuration matter for interpretation.
82.0% belongs to the whole pipeline. Review and exploration ran on Anthropic models inside the same trial, so that number cannot be attributed to GPT-5.6 Sol alone. The review-off run above is the isolation: 79.8%.
The served model was checked, not assumed. The CLI header echoes the requested model; the run notes verify gpt-5.6-sol at xhigh from the provider metadata attached to each assistant message.
Scoring rule and the replacement runs
The results sheet records one final outcome per task. Sixteen tasks were rerun on 26 August after their first attempt was classified as infrastructure-damaged; for those tasks the rerun result replaces the original.
Each square is one task. Hatched squares were scored on their replacement run. Hover for the task ID.
All task IDs and outcomes
bn-fit-modify— passedbuild-pmars— passedcancel-async-tasks— passedchess-best-move— passedcode-from-image— passedconfigure-git-webserver— passedconstraints-scheduling— passedcount-dataset-tokens— passedcrack-7z-hash— passeddb-wal-recovery— passeddistribution-search— passedextract-elf— passedfeal-differential-cryptanalysis— passedfeal-linear-cryptanalysis— passedfinancial-document-processor— passedfix-code-vulnerability— passedfix-git— passedfix-ocaml-gc— passedgcode-to-text— passedgit-leak-recovery— passedgit-multibranch— passedheadless-terminal— passedhf-model-inference— passedinstall-windows-3.11— passedkv-store-grpc— passedlarge-scale-text-editing— passedlargest-eigenval— passedllm-inference-batching-scheduler— passedlog-summary-date-ranges— passedmerge-diff-arc-agi-task— passedmodel-extraction-relu-logits— passedmodernize-scientific-stack— passedmteb-leaderboard— passedmteb-retrieve— passedmulti-source-data-merger— passednginx-request-logging— passedopenssl-selfsigned-cert— passedoverfull-hbox— passedpassword-recovery— passedpath-tracing— passedpath-tracing-reverse— passedpolyglot-c-py— passedpolyglot-rust-c— passedportfolio-optimization— passedprotein-assembly— passedprove-plus-comm— passedpypi-server— passedpytorch-model-cli— passedqemu-alpine-ssh— passedraman-fitting— passedregex-log— passedrstan-to-pystan— passedsanitize-git-repo— passedschemelike-metacircular-eval— passedsparql-university— passedsqlite-db-truncate— passedsqlite-with-gcov— passedtorch-tensor-parallelism— passedtune-mjcf— passedvulnerable-secret— passedwinning-avg-corewars— passedwrite-compressor— passedbreak-filter-js-from-html— passed, replacement runbuild-cython-ext— passed, replacement runbuild-pov-ray— passed, replacement runcaffe-cifar-10— passed, replacement runcobol-modernization— passed, replacement runcompile-compcert— passed, replacement runcustom-memory-heap-crash— passed, replacement rundna-insert— passed, replacement runmailman— passed, replacement runmake-mips-interpreter— passed, replacement runmcmc-sampling-stan— passed, replacement runfilter-js-from-html— failedgpt2-codegolf— failedmake-doom-for-mips— failedpytorch-model-recovery— failedqemu-startup— failedquery-optimize— failedregex-chess— failedreshard-c4-data— failedsam-cell-seg— failedtrain-fasttext— failedvideo-processing— failedadaptive-rejection-sampler— failed, replacement runcircuit-fibsqrt— failed, replacement rundna-assembly— failed, replacement runextract-moves-from-video— failed, replacement runtorch-pipeline-parallelism— failed, replacement run
This is neither all-attempt accuracy nor best-of-N. Replacement is applied where a rerun exists regardless of which result was better, and 11 of the 16 replacements passed while 5 failed. The rule still favours the system: only tasks whose first attempt was classified as infrastructure-damaged were rerun, and the sheet does not record those original outcomes. The sheet also counts 102 trials in total against 16 tasks marked as reruns, a discrepancy it does not resolve. Any comparison with a protocol that averages every trial must account for both points.
The 16 tasks scored on a replacement run are: adaptive-rejection-sampler, break-filter-js-from-html, build-cython-ext, build-pov-ray, caffe-cifar-10, circuit-fibsqrt, cobol-modernization, compile-compcert, custom-memory-heap-crash, dna-assembly, dna-insert, extract-moves-from-video, mailman, make-mips-interpreter, mcmc-sampling-stan, torch-pipeline-parallelism. Five of these are among the sixteen final failures.
Statistical precision
No repeated full sweep was run, so the sheet reports no interval. For a single binomial proportion, the Wilson 95% interval gives a sense of how much a result at this task count can move:
with , , .
The interval is illustrative. It assumes independent trials with a fixed per-task probability and does not model the replacement rule or the varying difficulty of the tasks. What it does show is scale: a gap of a few points between two single-sweep results on 89 tasks is well inside this range and is not, on its own, evidence that one system is better.
What the number can be compared with
Our site displays Claude Code at 78.9% and Codex at 78.4% beside our own result (79.8% with Sol at the time of this section, 95.1% with Opus 5.5 now). The table describes the Sol runs; the Opus 5.5 differences (refusals excluded, nine tasks on a later run) are listed in the update above. Those values are transcribed from the public tbench.ai 2.1 leaderboard and were not reproduced in this run. Four differences make the columns non-equivalent:
| Dimension | tbench.ai leaderboard | This run |
|---|---|---|
| Accuracy | successes ÷ all trials; reruns averaged in | one final outcome per task; reruns substituted |
| Trials per task | ≥ 5 for listed entries | 1, with 16 tasks scored on a replacement run |
| Cost | sum of every trial | last attempt per task only |
| Token count | input + output; cache tokens excluded | all four classes, cache reads included |
| Model | as submitted, one entry per model | 79.8%: GPT-5.6 Sol, Haiku for web fetch · 82.0%: + Opus review |
Under the leaderboard’s own rule — every attempt averaged rather than the replacement substituted — both runs would score lower — below 79.8% and 82.0% respectively — if the replaced attempts count as failures, and its cost would be scaled by the trial count. The comparability notes in the repository work through the adjustments.
A terminal benchmark also measures a narrow slice of what an agent product does. It says nothing about email, CRM work, browser tasks, permissions, or team workflows. For those, see TODO for AI vs Claude Code, vs Codex, or the comparison directory.
Cost of the review-on run: three models, four token classes
Cost is computed from provider usage reports on the last attempt of each task, including the sub-agent todos linked from each trial. Abandoned infrastructure-damaged attempts are not priced.
| Model | Role | Input | Output | Cache read | Cache write |
|---|---|---|---|---|---|
| GPT-5.6 Sol | main loop | 7.65M | 0.95M | 144.94M | 0 |
| Claude Opus 5 | review | ~0 | 1.74M | 24.31M | 2.68M |
| Claude Haiku 4.5 | explore, webfetch | 0.24M | 0.50M | 18.98M | 2.96M |
| Model | $/Mtok — in / out / cache read / cache write | Cost | Per task |
|---|---|---|---|
| GPT-5.6 Sol | 2 / 10 / 0.2 / 2.5 (promo) | $53.82 | $0.60 |
| Claude Opus 5 | 5 / 25 / 0.5 / 6.25 | $72.37 | $0.81 |
| Claude Haiku 4.5 | 1 / 5 / 0.1 / 1.25 | $8.32 | $0.09 |
| Total | $134.51 | $1.51 |
Three observations follow directly from the table.
- The review sub-agent costs more than the model being benchmarked — $72.37 against $53.82, from roughly 50 review calls, almost entirely Opus output and cache writes. Turning review off (the 79.8% run) cut the total to $46.87.
- Cache reads are 93% of all tokens sent. An agent loop re-sends its context every turn. A token count that reports only
input + output— as the leaderboard’s token column does — describes a small fraction of the traffic that providers bill for. - The price is a parameter, the tokens are the measurement. At GPT-5.6 Sol’s full list rate (4 / 20 / 0.4 / 5) the total becomes $188.33, or $2.12 per task instead of $1.51. Our own billing ledger is deliberately not the headline: it includes promotional discounts nobody outside can reproduce.
The sixteen failures
adaptive-rejection-sampler, circuit-fibsqrt, dna-assembly, extract-moves-from-video, filter-js-from-html, gpt2-codegolf, make-doom-for-mips, pytorch-model-recovery, qemu-startup, query-optimize, regex-chess, reshard-c4-data, sam-cell-seg, torch-pipeline-parallelism, train-fasttext, video-processing.
Eleven failed on the original sweep, five on a replacement run. The sheet records outcomes, not root causes, and this post does not reclassify any of them as infrastructure problems.
Reproducing the configuration
Public inputs:
- Results sheet — all 89 outcomes, token and price tables
- Task list
- Harbor adapter
- Token accounting script
The adapter calls a hosted backend and uses an account-configured agent named app. Model access, reasoning setting, sub-agent models and tool permissions live in that account, not in the repository, so a clone is a starting point rather than a hermetic replay. Use a dedicated benchmark account and Docker host with no unrelated credentials. A single-task invocation:
git clone https://github.com/todoforai/benchmarks.git
cd benchmarks && git checkout 4d37742c540801d0b2e10305446b02555a9fd2f1
cd terminal-bench
python3 -m venv .venv && . .venv/bin/activate && pip install -e .
: "${TODOFORAI_API_KEY:?Set a dedicated benchmark API key first}"
export TODOFORAI_API_KEY
harbor run \
-d "terminal-bench/terminal-bench-2-1" \
--agent-import-path "todoforai_tbench:TODOforAIHarborAgent" \
-m "openai:openai/gpt-5.6-sol" \
-i "terminal-bench/openssl-selfsigned-cert" \
--job-name "tb21-single-task" \
--max-retries 0 \
--yes -n 1The -m flag selects the main model; it does not set xhigh or configure sub-agents. Verify both from the returned message metadata before trusting a sweep.
For a run intended to be compared with the leaderboard: fix dependency versions and agent settings before starting, keep every attempt, declare the retry rule in advance, and report all-trial accuracy separately from any replacement-run diagnostic.