1 SCORE · 500 GRADED TASKS · 3 KINDS OF BOARD · 0 REQUIRED REFEREES
Eight chapters and a scorecard. One task and the test that grades it, 500 containers and the percentage they produce, the pull request that is the submission form. Then the four postures of provenance, an arena with no answer key, contests written after every training cutoff, and what state of the art actually claims. The scenes play themselves so watch or just scroll. are tappable. You are the submitter.
At 09:00:00 your coding agent is handed a repository pinned at commit d16bfe0 and 340 words of issue text. It is handed nothing else. By Friday there will be a row with your name on it, on three different boards, each saying something slightly different. This tutorial follows that row, and it carries a running meter: Σ graded attempts, counting every task, contest problem and human vote scored on your model’s behalf. It starts at one.
A is four fields: a repository URL, a base commit, the text of a GitHub issue, and a . Your agent is handed the first three. Instance astropy__astropy-12907 gives it the astropy repository pinned at one commit and 340 words describing a bug in how nested models compute separability. The fourth field stays sealed. It holds the tests written by the human who actually fixed this issue, and it is applied only after your agent has committed to an answer. That asymmetry is the whole design: the model is asked to reproduce a fix it cannot look up, and is marked against the real one.
Grading applies the test patch to whatever your agent produced and runs the suite. Two lists decide the outcome. names the tests that were failing before and must now pass, which is the proof the bug is fixed. PASS_TO_PASS names the tests that were already green and must stay green, which is the proof nothing else broke. Both lists must be satisfied. No rubric is consulted and no judge model is asked for an opinion, so two people running this instance on the same patch get the same answer every time.
Each instance ships as its own Docker image: an operating system, a Python version, every dependency pinned to the version that was current when the issue was filed, and the repository installed in editable mode. This is not incidental packaging. The environment is half the benchmark, because a test that passes under scipy 1.11.1 and fails under 1.12 turns a correct patch into a zero. Benchmarks that let dependencies float produce scores that drift with the package index, which is why every serious task set now pins to a digest.
The original SWE-bench holds 2,294 instances mined from merged pull requests, and a lot of them are unfair. OpenAI hired 93 experienced Python developers to read every one. They flagged 38.3% for problem statements too vague to solve from, and 61.1% for tests that would mark a valid solution wrong. In total 68.3% were filtered out, leaving the 500-instance called SWE-bench Verified, which is the one every launch post quotes. Keep that ratio in mind: the headline benchmark of agentic coding is the third of a benchmark that survived human review.
Resolved, or not resolved. There is no partial credit for a patch that fixes half the bug, and no bonus for a patch that is tidier than the human one. Run all 500 and you have 500 bits; 342 of them come back resolved. The is their mean, 342 ÷ 500 = 68.4%, and that is the entire content of the sentence “68.4% on SWE-bench Verified”. Every leaderboard percentage you have ever read is an average of a column like this one. The next chapter runs the other 499 and watches what it takes to produce the number.
Where instances come from. A miner walks a repository’s merged pull requests looking for ones that close an issue and touch test files. The code changes become the reference solution, the test changes become the test patch, and the issue body becomes the prompt. Nothing is written by hand, which is how you get 2,294 of them cheaply, and also how you get 1,794 unusable ones.
The issue text is the entire specification. There is no ticket description, no reproduction script and no maintainer to ask. If the human who fixed this bug had context from a mailing list thread, that context is gone. This is the single largest source of unfairness in mined benchmarks, and it is exactly what the 38.3% flag was measuring.
PASS_TO_PASS is the regression guard. Without it, a patch that deletes the failing assertion would score. With it, the agent has to leave the other 18 tests in the file alone. It is also the list most likely to fail for boring reasons: a warning promoted to an error, a timestamp in an assertion, a test that was already flaky before anyone got involved.
The variants. Beyond Verified there is Lite (a cheap 300-instance subset), Multimodal (front-end issues with screenshots), Multilingual (nine languages beyond Python) and Pro (longer, harder, commercially sourced tasks). They share a name and a repository layout, and nothing else.
What “solvable” hides. Many Verified instances have a reference fix of one or two lines. Screening removed the unfair tasks, not the easy ones, so a high resolve rate mixes genuine debugging with a fair number of one-line edits found by grep.
Five hundred containers start, and eight hours later one number comes out of them. Everything between those two facts is configuration: how many turns the agent gets, how many attempts it is allowed, and what happens when a build fails. The meter climbs from 1 to 500 here, which makes this the only chapter where Σ and the headline number count the same things.
The does the same six things 500 times: build the image, hand the agent the issue, let it work until it stops or hits the turn cap, take whatever patch it has at that moment, apply the test patch, record one bit. Notice who owns the turn cap. The harness does, not the model, so “how long is the agent allowed to think about this” is a benchmark setting rather than a model property. Eight hours later, 342 instances come back resolved and the run is over.
means one prediction per task, scored as submitted. means several attempts plus a step that picks among them, and on maths boards the same fork appears as cons@64 or maj@k. The gap between them is large enough to reorder a leaderboard, so SWE-bench makes you say which one you did: tags.system.attempts is either 1 or 2+, and the checklist in your pull request repeats the question in words. A table that omits this column is comparing a model that got one try against a model that got sixty-four.
Nothing here is deterministic. Sampling is stochastic, agents time out at different points, and container builds fail in ways that get counted as failures. Run the same model over the same 500 tasks five times with identical settings and the results land two points apart, which is why reported scores on this benchmark commonly differ by two to three points between groups who think they measured the same thing. Terminal-Bench draws the obvious conclusion and refuses single-run submissions: five runs, or no row.
Four boxes go into the pull request. You did pass@1. You did not read FAIL_TO_PASS or PASS_TO_PASS from the dataset. You did not use the hints field, which contains the maintainer comments from the original issue thread. Your agent either had no web access, or had it with steps taken to keep it from finding the commit that fixed the bug. Every one of these is checkable in principle from your trajectories and checked in practice by nobody, because the person confirming it is you.
Five hundred binary trials at a rate of 0.684 carry a binomial standard error of √(0.684 × 0.316 ÷ 500) = 0.021, or 2.1 points. So the honest way to write the result is 68.4% ± 2.1, and a rival at 70.1% is inside that interval. The leaderboard prints 68.4% and a rank. This is the first of three places in this tutorial where a benchmark reports a point estimate for a quantity it measured with visible noise, and it will matter again in chapter 05 and chapter 07. Σ is now 500.
Turn caps and token budgets. A typical agentic harness allows a few dozen tool calls and caps total tokens per instance. Raise the cap and the resolve rate rises with it, up to a point, then flattens. Since the cap is a harness setting rather than a model property, two groups running “the same model” can be running two quite different experiments.
Build failures count against you. If the image fails to build or the container runs out of disk, the instance records as unresolved. On a bad day this alone moves a run by a point, and it is invisible in the final percentage.
Timeouts are silent failures. An agent that is still working when the clock stops has its current patch taken, which is usually empty or half-written. The distinction between “got it wrong” and “ran out of time” disappears into the same zero, and only the trajectories preserve it.
best@1 is not pass@1. The metadata schema distinguishes them: best@1 means you ran the model several times and reported the single best run, while declaring one attempt per task. It scores like a multi-attempt method and reads like a single-attempt one, so the field exists to force the disclosure.
Re-running somebody else’s submission is not free. A full agentic pass over 500 instances is hours of wall clock and real API spend. That cost is the practical reason independent verification is rare, and it comes back in chapter 04.
The row says a model name, and that is a compression. What got measured was a model and the program driving it, and the program is worth about twenty points either way. This chapter takes the same weights through a second board with a second set of rules, which is where Σ picks up another 445 graded attempts and where the first two published numbers about your agent stop agreeing with each other.
Take the weights you just measured and wrap them in a different : fewer tools, a lower turn cap, no retry when the tests come back red. The resolve rate falls to 54.8%. Wrap them in a more generous one and it climbs to 74.2%. The model never changed. What the agent loop does between turns changed, and that was worth 19.4 points. Terminal-Bench responded to this by giving the scaffold its own column, so a row there names a model and an agent, because the pair is what got measured.
Terminal-Bench 2.0 puts an agent at a shell inside a container and gives it 89 curated tasks across 16 categories: software engineering, security, scientific computing, data science, debugging, games. Nearly a hundred contributors built it under Stanford and the Laude Institute, and they sized it deliberately, keeping the tasks hard enough that frontier performance stays under half. A submission runs the whole set five times through the Harbor framework and mails in the job directories. Σ climbs by 89 × 5 = 445, to 945.
Thinking budget, tool access, context window, retries, temperature, scaffold version. Six settings, each capable of moving a resolve rate by several points, and one of them occasionally appears beside a number, as the phrase “with tools”. That phrase is doing a lot of work: a model given a Python interpreter and a model given none are two different systems, and on maths benchmarks the gap between them is routinely larger than the gap between model generations. When a table lists one number per model, it has picked one configuration per model and thrown the rest away without saying so.
At least six task sets answer to the name. Full is the 2,294 mined instances. Verified is the human-screened 500. Lite is a cheap 300 for iteration. Multimodal is front-end issues with screenshots, Multilingual covers nine languages beyond Python, and Pro is longer commercially sourced work. Different difficulty, different , no shared scale. This is how three sites end up naming three different leaders on what sounds like one benchmark: they are quoting three different task sets, and none of them is lying.
Nothing stops you from tuning the scaffold to the shape of the task set. SWE-bench Verified is Python, pytest, one repository, one issue at a time, so a scaffold that assumes all four does well and then falls over on a benchmark where the agent has to install a package, read a manual page and drive a game. Rankings invert between task sets for exactly this reason. It is legal, it is cheap, and it is the main reason two boards disagree about which agent is best.
What Harbor standardizes. It fixes the container contract, the task format and the result schema, so two agents face an identical environment. It does not fix the agent: prompts, tool definitions, planning strategy and retry policy are yours. That line is deliberately drawn so the benchmark measures agents rather than devops.
Mini-agents versus frameworks. Some of the strongest SWE-bench submissions come from deliberately tiny scaffolds, a few hundred lines with bash and a file editor. The elaborate multi-agent frameworks often score lower, which is a useful result in itself and hard to see from a leaderboard that prints only the model name.
Why the columns split. Early leaderboards had one column and the community argued about attribution for two years. Boards that now separate model from agent did so because the alternative was unreadable: identical model names with twenty-point spreads and no way to tell why.
Version your scaffold like a dependency. If you publish a number, publish the commit of the agent that produced it. A resolve rate without a scaffold version cannot be reproduced by anybody, including you, six months later.
The cost of five runs. Requiring five passes over 89 agentic tasks multiplies the API bill by five. It is the right call statistically, and it is also a barrier to entry: the boards with the strictest protocols are the ones with the fewest independent submitters.
There are four places a benchmark table lives, and only one of them ever runs anything. What follows is the practical part: the addresses, the directory you commit, the fields you disclose, the badge that means less than it looks, and the six-field row you can actually defend. The meter stays at 945 the whole way through, which is the chapter’s point.
A lives in one of four places. The benchmark’s own authors keep one, usually a static site backed by a GitHub repository: swebench.com publishes the table, SWE-bench/experiments holds the submissions. A framework keeps one, like tbench.ai for Terminal-Bench, where the board and the runner ship together. Aggregators keep one, collecting hundreds of models into a single sortable table. And the lab keeps one, in the launch post, which is where most numbers appear first and where most readers meet them.
You open a pull request against SWE-bench/experiments adding one directory named evaluation/<split>/<date>_<model_name>. Inside it: all_preds.jsonl with one patch per instance, a logs/ folder holding patch.diff, report.json and test_output.txt for every instance, a trajs/ folder of , a metadata file and a README. Then you run python -m analysis.get_results and paste its output into the pull request body. The pull request is the application form, and one maintainer is the reviewer.
The metadata file is the disclosure. It carries the entry name, a link to the source if the system is open, and a link to a technical report or blog post. It also carries the contributors by name, the exact model identifiers used, whether the weights are open, whether the scaffold is open, and the attempts count. The report is not optional: submissions without adequate documentation get rejected. You also grant the maintainer push access to your branch. And since November 2025, the Verified and Multilingual boards narrowed further, taking submissions only from academic teams with peer-reviewed publications, so for those two the answer to “how do I apply” is now “publish a paper first”.
Watch the meter through this step: it does not move. An aggregator reads a percentage off a launch post, puts it in a row, sorts by it and adds a rank. A second aggregator copies the first. A news article copies the second. Four pages now carry the number, and it was executed exactly once, by the party it describes. This is the dominant mode of every table you have ever scrolled, and it is why a figure appearing in many places is not corroboration. Σ holds at 945.
The badges are not a standard. On SWE-bench, os_model and os_system say which halves of the system are open, and “checked” means a maintainer re-ran the submission on a random subset and got the same answer. On ARC Prize, “Verified” means something stricter: the run came from a trusted source, it is reproducible, and the public and semi-private scores agree within a stated tolerance. Two boards, two words, no shared definition, and most rows on most boards carry no badge at all.
If you publish a number yourself, six fields make it comparable to somebody else’s. Name the split. Name the scaffold and pin its commit. Declare the attempts. Publish the logs and the trajectories, which is the cheapest form of independence available to you. Say who ran it. Date it, because every figure in this field has a half-life. Any table missing one of the six is not comparable to another table, and the average launch post publishes two.
What the results script validates. analysis/get_results reads your predictions and logs, recomputes the resolve rate from the per-instance reports rather than trusting your summary, and cleans the directory into the layout the leaderboard renders from. If your claimed number and your logs disagree, this is where it surfaces.
Trajectories matter more than predictions. A patch tells a reviewer what your agent produced. A trajectory tells them how, including whether it opened a browser, read a test file it should not have, or got the answer on the first try because the fix was in its training data. Boards that require trajectories are asking for the only artifact that makes the honour-system checklist auditable.
Hosted leaderboards come and go. HuggingFace ran the Open LLM Leaderboard for years and retired it once the benchmarks under it saturated. A board is a maintained service, and when maintenance stops the table stops being a record of anything current.
Some submissions arrive by email. Terminal-Bench asks for five runs plus the job directories, sent to the maintainers for validation. It works, and it also tells you how much of this infrastructure is a handful of people reading attachments.
Rejection is normal. Missing report, missing logs, an implausible jump with no explanation, a scaffold nobody can inspect. The gate is one maintainer’s judgment, which is both the weakest part of the system and the reason the worst submissions never appear.
The default is that a lab runs the benchmark on its own model and publishes the result, and no one else touches it. Independence is not a property of the system; it is something a specific party pays for, one run at a time. This chapter sorts numbers by who made them, adds the first 500 attempts to Σ that you did not produce, and looks at two cases where the money and the task set came from the same place.
Walk the pipeline and ask who is required at each step. The benchmark authors built the task set. You ran the model. Your machine graded the output. You published the number. Who checks it? Nobody, and no rule anywhere says otherwise. There is no accrediting body for benchmark results, no audit requirement, no registration. Everything Σ has counted so far, all 945 attempts, is . That is not a scandal. It is the ordinary condition of the field, and it is the reason the rest of this chapter exists.
Sort every number you read by who produced it. Self-reported: the lab ran the benchmark on its own model. Author-run: the benchmark’s own authors ran it, which removes the incentive but not the scaffold question. Third-party: an outfit like Epoch AI, Artificial Analysis or Vals AI runs every model itself on one harness, so at least the differences between rows are real. And : the model is scored on tasks it can never have seen. Strength increases down that list, and so does cost, which is why the top of it is where almost everything sits.
A neutral harness re-runs the identical 500 tasks against the identical model and reports 64.9%. Yours said 68.4%. Neither party did anything wrong. Their scaffold has different tools and their API version is two weeks newer. Their retry policy gives up sooner, their sampling parameters differ, and refusals count as failures on their side. Σ climbs by 500 to 1,445, and this is the first entry on the meter that somebody other than you produced. A third-party number is not automatically the true one; it is a second measurement, and two measurements are how you find out the spread.
FrontierMath is a hard mathematics benchmark built by Epoch AI. In December 2024 Epoch disclosed that OpenAI had funded it, under an agreement that kept the funding quiet until the o3 announcement, the same announcement that set a record on it. OpenAI also had access to the problem set, covered by a verbal agreement not to train on it, and Epoch retained a separate holdout set. Epoch stated it had not been able to independently verify the result. Contributors said afterwards that they had not known who was paying. Nothing here requires bad faith to be a problem: the structure alone is enough to make the number hard to interpret.
The same shape appears at the centre of this tutorial. SWE-bench Verified, the 500-task everybody quotes, was assembled by OpenAI with 93 hired developers deciding which tasks were fair. Every lab now competes on the set one of them curated. Again, this is structural rather than an accusation, and the field’s answer to it is contractual rather than technical: terms on the APIs that see unpublished tasks, held-out splits nobody gets, and written no-train agreements where a verbal one used to do. Chapter 06 is where those measures get real teeth.
What zero-data-retention covers. It binds the API provider not to store or train on requests. It does not bind the model that already read the public half of the benchmark, and it does not survive a task set being published later. It is a control on the channel, not on the model.
Who pays for third-party runs. Somebody has to buy the tokens for every model on the board, repeatedly, as models ship. That bill is why neutral evaluators are few, why they are often funded by subscriptions or grants, and why their coverage of open models tends to lag their coverage of the big APIs.
Aggregate indices hedge the runner. If one evaluator’s harness has a quirk, an index combining many benchmarks dilutes it. It also dilutes the signal, and it introduces a new judgment call: the weights. Public, versioned methodology is the difference between an index and an opinion.
Logs are the cheap form of independence. You cannot make anyone re-run your submission, but you can publish every trajectory and let one curious reader find the instance where your agent read a file it should not have. Several corrections to published numbers have started exactly there.
“We could not verify this result.” When a benchmark author writes that sentence, it means the number on the board came from the model’s owner and the author had no access to reproduce it. It is the most informative sentence on many leaderboards and it is usually in a footnote.
What is being measured when there is no correct answer to compare against? Everything so far in this tutorial has had a grader that could be right: tests pass or they do not. The arenas throw that away and ask people which of two answers they prefer, then turn millions of those choices into a rating with an interval on it. Σ picks up 1,000 votes here, the first entries on the meter cast by humans.
You type a prompt. Two models answer side by side, names hidden. You pick A, B, tie, or both bad. In WebDev Arena the two answers are rendered web applications rather than text, so you are comparing running interfaces. There is no test patch here and no grader, because there is nothing to be right about: “write me a landing page for a bakery” has no reference solution. What accumulates is preference, and preference is a real signal about a different question than the one SWE-bench answers.
Ratings come from the , a win-probability model fit by maximum likelihood over the whole battle history at once. That is the difference from chess Elo, which updates incrementally and therefore depends on the order games were played: here a vote cast today can move a rating computed for a model from last year. The 95% intervals come from bootstrap resampling, and battles are reweighted so a heavily-voted pairing does not dominate. A model needs at least 1,000 votes before its rating stops drifting, and usually many more. Σ climbs by 1,000 to 2,445.
Interval width is driven mostly by vote volume. A model with 84,000 battles carries an interval of a few rating points; a model with 4,900 carries one of 30 or more. So a table where rank 3 sits 13 points above rank 7 is showing you two models it cannot distinguish, and the arena says so in a column most readers skip. Ranks are assigned by significant separation rather than raw order, which is why you see several models sharing a rank number. Sorting is not the same as knowing.
Response length and markdown formatting predict votes independently of whether the content is any good. Longer answers with headers and bold text win more often than short correct ones. The arena’s answer is , a second ranking that regresses those features out, and models move several places between the two boards. When you read an arena number, the first question is which of the two you are looking at, because they are measuring different things and both are published under the same brand.
Nothing gets uploaded. A provider hands over an API endpoint, preferably OpenAI-compatible, for a minimum of 30 days. During preview the model appears to voters under an anonymous codename, and results go back to the provider privately until the model is released. And the provider must confirm in writing that the previewed model is identical to the one they intend to ship. That clause has a date attached: in April 2025 a lab put a variant tuned for human preference on the board, verbose and heavy on emojis, while shipping a different model to the public. The rule was written afterwards, and models tuned specifically for the arena are now barred.
Preference and accuracy come apart on hard prompts. On a question the voter cannot check, a confident wrong answer beats a hedged right one. That failure mode is structural, it gets worse as prompts get harder, and it is the main reason arena rank and benchmark rank disagree at the top.
Category boards are much thinner. The overall board may have tens of thousands of votes per model; coding, maths, vision and WebDev subsets are slices of that. Same methodology, far wider intervals, and the same rank column presented with the same confidence.
What a tie vote does. Ties carry information about two models being close and are included in the fit rather than discarded. “Both are bad” is different again: it says something about the prompt, and pairs where both sides fail tell you where the frontier actually is.
Who is voting. The population is self-selected people who chose to visit an LLM arena. Their preferences are real and they are not a random sample of your users, so an arena rating is evidence about general appeal rather than about your product.
Why the arenas matter anyway. They are the only large public evaluation where the test set is generated fresh by users every day, so there is nothing to contaminate. Chapter 06 takes that idea further.
Baku, September 2025, the ICPC World Finals: twelve problems written for that room, on that day. No amount of scraping helps, because the problems did not exist when the training data was collected. This chapter covers the three ways benchmarks buy that property, by date- stamping, by holding sets back, and by holding live contests, and it adds the last 132 graded attempts to Σ.
Every fixed public task set decays: it gets published, it gets scraped, it ends up in a training corpus, and the score stops measuring reasoning. The structural fix is to keep writing new problems. LiveCodeBench continuously collects fresh problems from LeetCode, AtCoder and Codeforces and stamps each with its release date, so you can evaluate a model only on work published after its training cutoff and watch the score change as the window moves. LiveBench does the same with new questions every month. The date stamp is the whole mechanism.
The ICPC World Finals is the strongest version of that idea: problems written for one room on one day, with no prior existence anywhere. At the 2025 finals in Baku, an OpenAI reasoning system solved all twelve within the five-hour window. Gemini 2.5 Deep Think solved ten, and one of its ten was a problem that no human team solved at all. Σ climbs by 12. Twelve graded attempts is a tiny sample and it is also the least contaminable evidence in this entire tutorial, which is the trade you make when you move from task sets to contests.
The same year, at the AtCoder World Tour Finals in Tokyo, the heuristic track ran humans and an OpenAI entry against the same optimisation problem for ten hours. Przemysław Dębiak finished first with roughly 45.2 billion points to the model’s 43 billion, about 5% ahead. At the International Olympiad in Informatics that year a model placed sixth out of 330 entrants, at gold-medal level. Read those two results together: the model is inside the top ten of the strongest human field available, and the top of that field was still ahead in the track that rewards ten hours of judgment.
ARC-AGI-2 keeps three sets. The public set is open, for development. The set of 120 tasks scores frontier models through their APIs and is never published, which is what the zero-data-retention terms are for. The private set of 120 is competition only. All three are calibrated to be statistically similar, so a score on one is comparable to a score on another, which is the property that makes a held-out set useful rather than merely secret. The competition itself runs on Kaggle: twelve hours of wall clock, four NVIDIA L4 GPUs, and no internet access, so the tasks physically cannot leave the sandbox. Σ climbs by 120 to 2,577.
ARC Prize caps a single semi-private evaluation run at $10,000 and prints cost per task next to accuracy. That is not administrative tidiness. On a benchmark where you can spend more compute per task to search harder, accuracy without cost is not a result, because any score can be bought by anyone willing to pay enough per puzzle. The cap turns the benchmark into a question with two axes, and it is the clearest example in this tutorial of a benchmark author designing against the way their own measure would otherwise be gamed.
Contests are strong evidence and weak measurement. Twelve problems is a sample size that would embarrass any other chapter of this tutorial, and the conditions cannot be reproduced next Tuesday. What they buy is the one thing no task set can: certainty that nobody had seen the questions.
Codeforces ratings as a continuous axis. Rather than a percentage, a rating places a model in the distribution of human competitors. It moves with every contest, it has no ceiling, and it is directly interpretable by anyone who has competed. Chapter 07 comes back to metrics shaped like this.
Algorithmic versus heuristic tracks. Algorithmic problems have a correct answer and reward finding it fast. Heuristic problems have no known optimum and reward ten hours of incremental improvement. Models have been much stronger at the first than the second, which is why the AtCoder result reads differently from the ICPC one.
Offline sandboxes as a control. No internet means no retrieval of a published solution and no phoning home with the task text. It also rules out the largest models, since whatever runs has to fit on four GPUs, so the sandbox selects for a different kind of entry than an API leaderboard does.
Rolling benchmarks are expensive to keep. Somebody has to source, validate and containerise new problems forever, and re-score every model on every window. A rolling benchmark that stops rolling becomes a fixed benchmark with a misleading name.
State of the art asserts one thing: the highest published score on a named benchmark, on a named split, under a named protocol, as of a named date. Everything difficult about the phrase comes from how easily those four qualifiers fall off. Σ stops moving here at 2,577, because aggregating and ranking grade nothing new; they only rearrange what the previous seven chapters produced.
asserts exactly one thing: the highest published score on a named benchmark, on a named split, under a named protocol, as of a named date. Four qualifiers. A headline keeps the model name and drops all of them, and the sentence still reads fine, which is why two “state of the art on coding” claims from the same week routinely turn out to be about different task sets measured different ways. Attach the four and most disagreements about which model is best dissolve into which cell somebody was quoting.
For a decade the nearest thing to a canonical registry was Papers with Code: 9,327 benchmark leaderboards and 79,817 links between papers and their implementations, free to use, the place you went to check whether a claimed record was actually a record. Meta shut it down in late July 2025 without notice. The data is frozen on GitHub and HuggingFace, the domain redirects to a trending-papers page, and nothing has replaced it. So when somebody says a model is state of the art, there is now no shared place to look it up, and the claim rests on whoever is repeating it.
A benchmark is useful while it has . MMLU launched in 2020 with the best models near 43% and now sits above 90% for everything at the frontier, which is : it can no longer tell two good models apart. Humanity’s Last Exam was built as the reply, 2,500 questions from nearly a thousand experts across more than 500 institutions in 50 countries. At launch in early 2025 the best score was 2.7%. A year later it is around a quarter. The evals tutorial covers what saturation does to your own test suite; here the point is narrower. Check headroom before you read a rank.
Draw the top six rows of a 500-task board with the ±2.1 error bar from chapter 01 and they form one blur. First place is 4.2 points above sixth, every interval overlaps its neighbours, and the ordering would shuffle if everyone re-ran tomorrow. The arena has the same shape for a different reason: intervals there are set by vote volume, and the thinly-voted rows carry ±30 rating points. Rank is the least informative column on either page, and it is the one every summary quotes.
Two responses to the ceiling problem. Aggregate indices, like Epoch AI’s capabilities index, combine many benchmarks into one versioned scale, so no single saturated exam decides the ordering. And continuous axes measure something with no maximum: METR’s 50% is the human-expert task length a model completes half the time, measured on over a hundred software, machine-learning and cybersecurity tasks against timed human baselines. It has doubled roughly every seven months since 2019, faster since 2024, and METR states that measurements above 16 hours are unreliable with the current task suite. A duration cannot reach 100%, so this metric does not expire the way a percentage does. It just gets harder to measure.
An index needs a version number. Combining benchmarks means choosing weights, and changing the weights changes the ordering. An index with public, versioned methodology is a measurement; one that reweights without saying so is an opinion with a chart.
Doubling times are fragile. A trend fitted over six years is sensitive to how tasks are sampled, how human baselines are timed, and which models count as frontier. Treat “doubling every N months” as a description of the data collected so far rather than a law.
Report the score with its cost and date. Three fields together are a result: what it scored, what it cost per task, and when. Any one of them alone is decoration, and cost is the one most often missing precisely because it is the one that constrains the claim.
Negative headroom exists too. A benchmark where every model scores near zero also fails to discriminate, and it is easy to mistake for a hard benchmark that is working. The useful window is the middle, which is exactly why Terminal-Bench sized itself to keep frontier scores under half.
Benchmarks have half-lives now. MMLU took about four years to saturate. Humanity’s Last Exam has moved an order of magnitude in one. Plan for any benchmark you adopt to stop discriminating within roughly two years, and write your reporting so that swapping it out does not invalidate your history.
The cell from the hero, with everything the cell does not say written underneath it. The right margin says which chapter earned each line. Of the 2,577 graded attempts on the meter, you produced 945, a third party produced 620, humans cast 1,000 votes, and a contest posed 12 problems. The headline quotes 500 of them.