Benchmarks · no Elo

TODO Bench

Real-life todos, done end to end. Every model gets the same fixed inputs and nothing leaves the sandbox: emails and posts stay drafts. Upvote the tasks you want us to run first.

scoring: proposals, upvotes · todo-bench/

Triage an inbox

A seeded test mailbox of 200 emails: label them and draft replies (drafts only) for the ones that need an answer.

Collect a quarter's invoices

Same mailbox: find every invoice, save the PDFs into one folder, list them in a sheet with totals. One right answer.

Reconcile a bank statement

A fixed month of transactions (CSV) and invoice PDFs: list what is missing or does not add up.

Find 20 B2B leads

From a fixed customer profile: 20 matching companies with a decision maker and email, into a sheet. Checked by hand.

Schedule a meeting

Four fixed calendars across time zones: find the slot and draft the invite. Nothing gets sent.

Plan a business trip

Cheapest sensible flight and hotel for given dates and rules, stopping before payment.

Compare competitor pricing

Snapshots of 5 pricing pages, before and after: report every change.

Launch posts from a product page

Ready-to-post drafts for LinkedIn, X and Facebook in a given brand voice. Drafts only, judged side by side.

Write an SEO article

Keyword research, outline and a 1,500-word article for a given site, judged side by side.

Build a landing page

One-paragraph brief → a working page with a signup form, run locally and screenshotted.

Fix a failing build

A repo with a red CI: find the cause and fix it until the tests pass.

Market size report

Size a market with sourced numbers in a short report; every citation gets checked.

Each model runs in its own sandbox through the TODO for AI agent — identical prompt, no view of the others. We publish the raw outputs, what they cost and how long they took. No Elo, nothing fitted to a curve. Upvotes are a separate reader signal and never change the order. Reproducible from the benchmarks repo.