AI benchmarks:
what the numbers actually measure

A benchmark is a fixed set of tasks, a program that puts a model through them, and a rule that marks each answer. Nothing else. The task set is written once and then reused, which is what makes two models comparable at all. The program that runs it is called the harness. The rule that marks the answers is the grader. Change any one of the three and the published number changes, which is why most of this tutorial is about telling them apart.

Here is one real task from SWE-bench, the best known coding benchmark. A task is four fields: a repository, the commit it is pinned to, the text of a GitHub issue, and a set of tests. The model is given the first three. Instance astropy__astropy-12907 gives it the astropy repository at one commit, plus about 340 words describing a bug in how nested models compute separability. The tests are withheld until the model has answered. They are the tests the human who really fixed this bug wrote, taken from the pull request that closed the issue.

from datasets import load_dataset

# The task set is a public download. Anyone can read every question in it.
tasks = load_dataset("princeton-nlp/SWE-bench_Verified", split="test")
task = next(t for t in tasks if t["instance_id"] == "astropy__astropy-12907")

task["repo"]               # 'astropy/astropy'
task["base_commit"]        # 'd16bfe0...', the commit the repo is pinned to
task["problem_statement"]  # the text of the GitHub issue, about 340 words
task["patch"]              # the fix a human really merged. Never shown.
task["test_patch"]         # the tests that grade the answer. Never shown.
task["FAIL_TO_PASS"]       # tests that must go from failing to passing
task["PASS_TO_PASS"]       # tests that were already passing and must stay so

# The model is handed the first three fields, and nothing else.
prompt = f"""Repository {task['repo']} at commit {task['base_commit']}.
Fix the issue below. Reply with a unified diff.

{task['problem_statement']}"""

Three kinds of grader exist, and every benchmark you will read about uses one of them. An answer key compares the answer to a stored correct one, either by string match or, as in SWE-bench, by running a test suite. A person compares two answers and says which is better, which is what arenas do. A second model reads the answer against a written rubric and returns a verdict, called an LLM judge. The first needs a question with a known answer. The second and third are what you reach for when no such answer exists.

def grade_with_answer_key(answer: str, expected: str) -> bool:
    """Multiple choice, maths, anything with one correct string."""
    return normalize(answer) == normalize(expected)


def grade_with_tests(repo: Path, test_patch: str, must_pass: list[str]) -> bool:
    """SWE-bench. The answer key is a test suite instead of a string."""
    apply_patch(repo, test_patch)
    return run_pytest(repo, must_pass).returncode == 0


def grade_with_human_vote(answer_a: str, answer_b: str) -> str:
    """Arenas. No key exists, so a person picks. "a", "b", "tie" or "both bad"."""
    return show_side_by_side_and_wait(answer_a, answer_b)


def grade_with_model_judge(answer: str, rubric: str) -> bool:
    """A cheaper stand-in for the person, and it needs its own accuracy check."""
    verdict = judge_model.complete(JUDGE_PROMPT.format(rubric=rubric, answer=answer))
    return verdict.strip().lower().startswith("pass")

Each task produces one result. On SWE-bench that result is a single bit: the tests passed, or they did not. There is no partial credit for a patch that fixes half the bug, and no bonus for a patch tidier than the human one. Run all 500 tasks and you have 500 bits, of which 342 come back resolved. The resolve rate is their mean, 342 divided by 500, or 68.4%. That average is the entire content of the sentence “68.4% on SWE-bench Verified”.

Task sets give the model a job to do in a controlled environment and check the result, as SWE-bench and Terminal-Bench do. Question banks ask hard exam questions with known answers, as MMLU, GPQA and Humanity’s Last Exam do. Arenas show two answers to a person and record which one they prefer. Live contests put models into competitions written on the day, such as the ICPC world finals. The rest of this tutorial walks through all four, because the same word “benchmark” covers them all and they mean different things.

Grading applies the withheld tests, called the test patch, to whatever the model produced, and runs them. Two lists decide the outcome. FAIL_TO_PASS names the tests that were failing before and must now pass, which is the proof the bug is fixed. PASS_TO_PASS names the tests that were already passing and must stay that way, which is the proof nothing else broke. Both lists must be satisfied. No rubric is consulted and no model is asked for an opinion, so two people grading the same patch get the same answer.

def resolved(task, model_patch: str) -> bool:
    """One task in, one bit out. No partial credit anywhere in this function."""
    with docker_container(task["instance_id"]) as repo:   # pinned OS and versions
        if not apply_patch(repo, model_patch):
            return False                                  # the patch did not apply

        apply_patch(repo, task["test_patch"])             # sealed tests arrive now
        f2p = json.loads(task["FAIL_TO_PASS"])
        p2p = json.loads(task["PASS_TO_PASS"])

        report = run_pytest(repo, f2p + p2p)
        return all(report[test] == "PASSED" for test in f2p + p2p)

Each task ships as its own Docker image: an operating system, a Python version, every dependency pinned to the version that was current when the issue was filed, and the repository installed in place. This is not incidental packaging. The environment is half the benchmark, because a test that passes under scipy 1.11.1 and fails under 1.12 turns a correct patch into a zero. Benchmarks that let dependencies float produce scores that drift with the package index, which is why serious task sets pin every version.

Tasks are mined automatically, and a lot of them are unfair. The original SWE-bench holds 2,294 of them, taken from merged pull requests. OpenAI paid 93 experienced Python developers to read every one. They flagged 38.3% for problem statements too vague to solve from, and 61.1% for tests that would mark a valid solution wrong. In total 68.3% were filtered out, leaving the 500-task split called SWE-bench Verified. That is the one every launch post quotes. The best known benchmark of agentic coding is the third of a benchmark that survived paid human review.

“SWE-bench” names at least six different task sets. Full is the original 2,294. Verified is the screened 500. Lite is a cheap 300-task subset. Multimodal covers front-end issues with screenshots, Multilingual covers nine languages beyond Python, and Pro holds longer commercially sourced tasks. Their scores are not comparable to each other. A number quoted without its split is missing the piece that says which exam was sat.

A task is not always a GitHub issue. On Terminal-Bench it is an instruction, a container and a test script: install this package, recover this corrupted archive, get this server responding. On GPQA it is a multiple-choice question written by a PhD holder, hard enough that expert non-specialists with web access score around 34%. On ARC-AGI it is a handful of coloured grids showing a rule, plus one grid to complete. Different subjects, same three parts from chapter 00.

The harness does the same six things 500 times: build the image, hand the model the issue, let it work until it stops or hits the turn cap, take whatever patch it has at that moment, apply the withheld tests, record one bit. Notice who owns the turn cap. The harness does, not the model, so “how long is the model allowed to work on this” is a benchmark setting rather than a model property. Some hours later, 342 tasks have come back resolved and the run is over.

results = {task["instance_id"]: resolved(task, patches[task["instance_id"]])
           for task in tasks}

resolve_rate = sum(results.values()) / len(results)   # 342 / 500

print(f"{resolve_rate:.1%} on SWE-bench Verified")    # 68.4% on SWE-bench Verified

Nothing about running a benchmark is privileged. The task set is a public download, and the harness that grades it is a package the benchmark’s own authors publish. Install it, point it at a file of answers, and it builds the containers and grades them on your machine. Grading is cheap: it costs disk, Docker and a few hours. Producing the answers is the expensive half, because that means 500 model runs, and on a frontier model that is a real bill. This is worth knowing because it makes third-party checking possible, which chapter 05 is about.

pip install swebench    # the harness, published by the benchmark's own authors
docker info             # required. One container image per task, tens of GB.

# preds.jsonl holds one line per task: {"instance_id": ..., "model_patch": ...}
python -m swebench.harness.run_evaluation \
  --dataset_name princeton-nlp/SWE-bench_Verified \
  --predictions_path preds.jsonl \
  --run_id my_first_run \
  --max_workers 8

# Grading the 500 tasks costs nothing but time and disk.
# Producing preds.jsonl is the expensive half: that is 500 agent runs.

pass@1 means one answer per task, scored as submitted. best@k means several attempts plus a step that picks among them, and on maths boards the same idea appears as cons@64 or maj@k. The gap between them is large enough to reorder a leaderboard, so SWE-bench makes submitters say which one they did: tags.system.attempts is either 1 or 2+, and the checklist in the pull request repeats the question in words. A table that omits this column may be comparing a model that got one try against a model that got sixty-four.

import math


def pass_at_k(n: int, c: int, k: int) -> float:
    """Chance that at least one of k attempts passes, given c of n did.

    From the Codex paper. Estimating it this way, instead of asking whether any
    of k tries happened to pass, removes the bias in a small sample.
    """
    if n - c < k:
        return 1.0
    return 1.0 - math.prod((n - c - i) / (n - i) for i in range(k))


pass_at_k(n=10, c=1, k=1)   # 0.10, one attempt allowed
pass_at_k(n=10, c=1, k=8)   # 0.80, eight attempts at the same task

Nothing here is deterministic. Sampling is random, models time out at different points, and container builds fail in ways that get counted as failures. Run the same model over the same 500 tasks five times with identical settings and the results land a couple of points apart. That is one reason reported scores commonly differ by two to three points between groups who think they measured the same thing. Terminal-Bench draws the obvious conclusion and refuses single-run submissions: five runs, or no row.

Four boxes go into the pull request. The run was pass@1. Nobody read FAIL_TO_PASS or PASS_TO_PASS from the dataset. Nobody used the hints field, which holds the maintainer comments from the original issue thread. The model either had no web access, or had it with steps taken to stop it finding the commit that fixed the bug. Each of these is checkable in principle from the recorded run and checked in practice by nobody, because the person confirming it is the person who ran it.

Five hundred yes-or-no trials at a rate of 0.684 carry a standard error of 2.1 points, which puts the 95% interval at roughly 64.3% to 72.5%. So a rival reporting 70.1% is inside the interval of a model reporting 68.4%, and the difference between them is not evidence of anything. The leaderboard prints 68.4% and a rank. This is the first of three places in this tutorial where a benchmark reports a single number for a quantity it measured with visible noise, and it comes back in chapters 06 and 08.

import math

n, resolved_count = 500, 342
p = resolved_count / n                    # 0.684

se = math.sqrt(p * (1 - p) / n)           # 0.0208, so 2.1 points
lo, hi = p - 1.96 * se, p + 1.96 * se     # 0.643 to 0.725

print(f"{p:.1%}, 95% interval {lo:.1%} to {hi:.1%}")
# 68.4%, 95% interval 64.3% to 72.5%
# A rival reporting 70.1% is inside that interval. The table still ranks it above.

Take one model and wrap it in a scaffold with fewer tools, a lower turn cap and no retry when the tests come back red. The resolve rate falls to 54.8%. Wrap the same weights in a more generous one and it climbs to 74.2%. The model did not change. What the agent loop does between turns changed, and that was worth 19.4 points. Terminal-Bench responded by giving the scaffold its own column, so a row there names a model and an agent, because the pair is what got measured.

def solve(task, model, tools, max_turns: int = 40) -> str:
    """The harness around the model. Every argument here changes the score."""
    messages = [{"role": "user", "content": build_prompt(task)}]

    for _ in range(max_turns):
        reply = model.chat(messages, tools=tools)
        if not reply.tool_calls:
            return extract_diff(reply.content)
        messages += run_tools(reply.tool_calls)

    return ""   # out of turns. An empty patch is graded as not resolved.


TOOLS = ["read_file", "grep", "edit", "run_tests"]
# Take run_tests away and the same model cannot check its own work before
# answering. Nothing about the weights changed, and the score falls.

Terminal-Bench 2.0 puts a model at a shell inside a container and gives it 89 curated tasks across 16 categories: software engineering, security, scientific computing, data science, debugging, games. Nearly a hundred contributors built it under Stanford and the Laude Institute, and they sized it deliberately, keeping the tasks hard enough that frontier performance stays under half. A submission runs the whole set five times through the Harbor framework and sends in the job directories.

Thinking budget, tool access, context window, retries, temperature, scaffold version. Six settings, each able to move a resolve rate by several points. One of them occasionally appears beside a number, as the phrase “with tools”. That phrase is doing a lot of work. A model given a Python interpreter and a model given none are two different systems, and on maths benchmarks the gap between them is routinely larger than the gap between model generations. A table that lists one number per model has picked one configuration per model and thrown the rest away without saying so.

Nothing stops a team from tuning the scaffold to the shape of the task set. SWE-bench Verified is Python, pytest, one repository, one issue at a time, so a scaffold that assumes all four does well there and then falls over on a benchmark where the model has to install a package, read a manual page and drive a game. Rankings invert between task sets for exactly this reason. It is allowed, it is cheap, and it is the main reason two boards disagree about which system is best.

Four separate jobs sit behind a published score. Someone writes the tasks and the grading rule. Someone runs a model against them. Someone publishes a table and decides which rows go on it. Someone repeats the number elsewhere. The authors are mostly universities and non-profits: SWE-bench came out of Princeton with collaborators at Stanford and Chicago, Terminal-Bench out of Stanford and the Laude Institute, ARC-AGI from the ARC Prize Foundation, FrontierMath from Epoch AI. The runner is usually the lab that made the model. Those two facts together explain most of this tutorial.

A leaderboard lives in one of four places. The benchmark’s authors keep one: swebench.com publishes the table and SWE-bench/experiments holds the submissions behind it. A framework keeps one, like tbench.ai for Terminal-Bench, where the board and the runner ship together. Aggregators keep one, collecting hundreds of models into a single sortable table. And the lab keeps one, in its launch post, which is where most numbers appear first and where most readers meet them.

For a code benchmark, applying means opening a pull request against SWE-bench/experiments with one directory named evaluation/<split>/<date>_<model_name>. Inside it: all_preds.jsonl with one answer per task, a logs/ folder holding patch.diff, report.json and test_output.txt for every task, a trajs/ folder of trajectories, a metadata file and a README. Then run python -m analysis.get_results and paste its output into the pull request. The application form is a pull request, and the reviewer is one maintainer.

Arenas work the other way round. Nothing is uploaded, because there is nothing to grade offline. A provider hands over an API endpoint, preferably OpenAI-compatible, for at least 30 days, and the arena spends those days sending it real user prompts. During the preview the model appears to voters under an anonymous codename, and results go back to the provider privately until release. The provider must confirm in writing that the previewed model is the one they intend to ship. That clause has a date attached: in April 2025 a lab put a variant tuned for human preference on the board while shipping a different model to the public. Models tuned specifically for the arena are now barred.

The metadata file is the disclosure. It carries the entry name, a link to the source if the system is open, and a link to a technical report. It also carries the contributors by name, the exact model identifiers used, whether the weights are open, whether the harness is open, and the attempts count. The report is not optional, and submissions without adequate documentation are rejected. Since November 2025 the Verified and Multilingual boards narrowed further, taking submissions only from academic teams with peer-reviewed publications, so for those two splits the answer to “how do I apply” is now “publish a paper first”.

An aggregator reads a percentage off a launch post, puts it in a row, sorts by it and adds a rank. A second aggregator copies the first. A news article copies the second. Four pages now carry the number, and it was executed exactly once, by the party it describes. This is how most tables anyone has ever scrolled are built. It is also why a figure appearing in many places is not corroboration: the copies are downstream of one run, and none of the copying parties owns a container.

The badges are not a standard. On SWE-bench, os_model and os_system say which halves of the system are open, and “checked” means a maintainer re-ran the submission on a random subset and got the same answer. On ARC Prize, “Verified” means something stricter: the run came from a trusted source, it is reproducible, and the public and semi-private scores agree within a stated tolerance. Two boards, two words, no shared definition, and most rows on most boards carry no badge at all.

Walk the pipeline and ask who is required at each step. The authors built the task set. The lab ran the model. The lab’s own machine graded the output. The lab published the number. Who checks it? Nobody, and no rule anywhere says otherwise. There is no accrediting body for benchmark results, no audit requirement and no registration. Almost every number in circulation is self-reported. That is not a scandal, it is the ordinary condition of the field, and it is the reason the rest of this chapter exists.

Sort every number you read by who produced it. Self-reported: the lab ran the benchmark on its own model. Author-run: the benchmark’s own authors ran it, which removes the incentive but not the harness question. Third-party: an outfit such as Epoch AI, Artificial Analysis or Vals AI runs every model itself on one harness, so at least the differences between rows are real. And held-out: the model is scored on tasks it can never have seen. Confidence increases down that list, and so does cost, which is why almost everything sits at the top of it.

A neutral group re-runs the identical 500 tasks against the identical model and reports 64.9% where the lab reported 68.4%. Neither party did anything wrong. Their harness has different tools and their API version is two weeks newer. Their retry policy gives up sooner, their sampling settings differ, and refusals count as failures on their side. A third-party number is not automatically the true one. It is a second measurement, and two measurements are how anyone finds out the spread.

FrontierMath is a hard mathematics benchmark built by Epoch AI. In December 2024 Epoch disclosed that OpenAI had funded it, under an agreement that kept the funding quiet until the o3 announcement, the same announcement that set a record on it. OpenAI also had access to the problem set, covered by a verbal agreement not to train on it, and Epoch retained a separate holdout set. Epoch stated it had not been able to verify the result independently. Contributors said afterwards that they had not known who was paying. Nothing here requires bad faith to be a problem. The structure alone is enough to make the number hard to interpret.

The same shape sits at the centre of this tutorial. SWE-bench Verified, the 500-task split everybody quotes, was assembled by OpenAI, with 93 hired developers deciding which tasks were fair. Every lab now competes on a set that one of them curated. This is structural rather than an accusation, and the field’s answer to it is contractual rather than technical: zero-data-retention terms on the APIs that see unpublished tasks, held-out splits nobody gets, and written no-train agreements where a verbal one used to do. Chapter 07 is where those measures get real teeth.

Publishing a number of your own takes six fields to make it comparable to somebody else’s. Name the split. Name the harness and pin its commit. Declare the attempts. Publish the logs and the trajectories, which is the cheapest form of independence available to anyone. Say who ran it. Date it, because every figure in this field has a half-life. A table missing one of the six is not comparable to another table, and the average launch post publishes two.

A visitor types a prompt. Two models answer side by side with their names hidden, and the visitor picks A, B, tie, or both bad. In WebDev Arena the two answers are rendered web pages rather than text, so what is being compared is two running interfaces. There is no hidden test here and no grader, because there is nothing to be right about: “build me a landing page for a bakery” has no reference solution. What accumulates is preference, which is a real signal about a different question than the one SWE-bench answers.

The voters are whoever opens the site. They are not paid, not screened for expertise and not identified, and their votes are the entire dataset. That sounds fragile, and the arena’s answer is not to trust any single vote. Model names are hidden, so brand preference cannot operate. Which model appears on the left is random, so position preference averages out. Duplicate and automated voting is filtered. The remaining bias is real and structural: on a question the voter cannot check, a confident wrong answer beats a hedged right one, and no amount of blinding fixes that.

The other way to use people is to hire them. SWE-bench Verified exists because OpenAI paid 93 experienced Python developers to screen tasks. Humanity’s Last Exam was assembled from questions written by close to a thousand subject experts, with prizes paid per accepted question. Hired annotators are screened, briefed with a written rule and measured against each other for agreement, which makes their labels usable as an answer key. They also cost money per item, so there are hundreds of them rather than millions. Arenas buy scale; hired experts buy reliability.

Ratings come from the Bradley-Terry model, a win-probability model fitted over the whole battle history at once. That is the difference from chess Elo, which updates incrementally and therefore depends on the order the games were played. Here a vote cast today can move a rating computed for a model last year. The intervals come from refitting on resampled battles, and battles are reweighted so one heavily-voted pairing does not dominate. A model needs at least a thousand votes before its rating stops drifting, and usually many more.

import numpy as np
from sklearn.linear_model import LogisticRegression

# One row per battle: +1 for the model on the left, -1 for the one on the right.
X = np.zeros((len(battles), n_models))
y = np.zeros(len(battles))
for i, (left, right, winner) in enumerate(battles):
    X[i, left], X[i, right] = 1, -1
    y[i] = 1.0 if winner == "left" else 0.0

fit = LogisticRegression(fit_intercept=False).fit(X, y)
ratings = 400 / np.log(10) * fit.coef_[0] + 1000   # rescaled to look like Elo

# The interval comes from refitting on resampled battles, over and over.
draws = [fit_ratings(resample(battles)) for _ in range(100)]
lo, hi = np.percentile(draws, [2.5, 97.5], axis=0)

Interval width is driven mostly by vote volume. A model with 84,000 battles carries an interval of a few rating points; a model with 4,900 carries one of 30 or more. So a table where rank 3 sits 13 points above rank 7 is showing two models it cannot tell apart, and it says so in a column most readers skip. Ranks are assigned by significant separation rather than raw order, which is why several models share a rank number. Formatting also moves votes: longer answers with headers and bold text win more often, so the arena publishes a second board with style control applied, and models move several places between the two.

The third grader from chapter 00 is a model. Give one the question, the answer and a written rubric, and it returns a verdict in seconds for a fraction of a cent. That is the only way to grade thousands of open-ended answers on a schedule, and it is how most private evaluation is done. It also has known failure modes: judges prefer whichever answer they read first, prefer longer answers, and prefer text from their own model family. The fixes are mechanical. Ask twice with the sides swapped and keep only the verdicts that agree with themselves, then measure the judge against human labels before trusting it. The evals tutorial covers that measurement in depth.

def judge(question: str, answer_a: str, answer_b: str) -> str:
    """Ask a model which answer is better. Returns "a", "b" or "tie"."""
    reply = judge_model.complete(JUDGE_PROMPT.format(q=question, a=answer_a, b=answer_b))
    return parse_verdict(reply)


def judge_both_ways(question: str, answer_a: str, answer_b: str) -> str | None:
    """Judges favour whichever answer they read first, so ask it twice."""
    first = judge(question, answer_a, answer_b)
    second = judge(question, answer_b, answer_a)     # same pair, sides swapped
    return first if first == flip(second) else None  # None: it disagreed with itself


# A judge is worth using only once you know how often it matches people.
agreement = sum(judge_both_ways(*pair) == label for pair, label in labelled) / len(labelled)

A benchmark has to be public to be shared, and a model is trained on a scrape of the public web. Those two facts collide. SWE-bench Verified is a dataset anyone can download, reference fixes included. MMLU and GPQA questions sit in repositories, papers, blog posts and tutorials with the answers beside them. If any of that was in the training data, part of the score is recall rather than reasoning. This is called contamination, and it needs nobody to cheat. It is the default outcome of publishing an exam on the internet that the candidates read.

Authors stamp a canary string into the file so corpus builders can filter it out, which helps only if they bother and does nothing about the copies reposted without it. Labs run their own filter, usually dropping any training document that shares a run of about 13 words with a test question, but that needs the training corpus and only the lab has it. So the public measurement is indirect: write new questions matched to the old ones for style and difficulty, and see which models drop. Scale AI did this with GSM1k, 1,250 fresh grade-school maths problems mirroring GSM8K. Some families fell by as much as 13 points, notably Mistral and Phi, while the frontier models from OpenAI, Anthropic and Google showed little gap.

def overlaps_training_data(question: str, corpus_ngrams: set[str], n: int = 13) -> bool:
    """The usual filter: one shared run of 13 words is treated as a leak."""
    words = question.split()
    runs = (" ".join(words[i : i + n]) for i in range(len(words) - n + 1))
    return any(run in corpus_ngrams for run in runs)


# That filter needs the training corpus, which only the lab that trained the
# model has. So the public measurement is indirect: rebuild the benchmark from
# scratch with new questions of the same difficulty, and compare.
gap = accuracy(model, gsm8k) - accuracy(model, gsm8k_rebuilt)

# Or keep only the questions published after the model stopped reading.
fresh = [p for p in problems if p.release_date > MODEL_TRAINING_CUTOFF]

LiveCodeBench records the publication date of every problem it collects, so a model can be scored only on problems that appeared after its training data was gathered. Nothing is hidden and the filter is one comparison per problem. The cost is that the usable set shrinks as models get newer, so it has to be refilled continuously from live sources. A rolling benchmark is a subscription rather than a file, and somebody has to keep paying for it.

ARC-AGI keeps three sets. A public one anyone may train against, a semi-private one behind the published leaderboard, and a fully private one that never leaves the machines it is graded on. Runs against the private set happen offline, on hardware with no internet connection, so the tasks cannot leak through an API. Where a model can only be reached over the network, the fallback is contractual: zero-data-retention terms saying the provider will not store or train on what it is sent. One of those is a physical guarantee. The other is a promise.

A live contest is the strongest version of the idea: problems written for one room on one day, with no prior existence anywhere. No amount of scraping helps, because the problems did not exist when the training data was collected. At the 2025 ICPC World Finals in Baku, an OpenAI reasoning system solved all twelve problems inside the five-hour window, and Gemini 2.5 Deep Think solved ten, one of which no human team solved. Twelve problems is a tiny sample. It is also the least contaminable evidence in this tutorial, which is the trade you make when you move from task sets to contests.

The same year, at the AtCoder World Tour Finals in Tokyo, the heuristic track ran humans and an OpenAI entry against the same optimisation problem for ten hours. Przemysław Dębiak finished first with roughly 45.2 billion points to the model’s 43 billion, about 5% ahead. At the International Olympiad in Informatics that year a model placed sixth out of 330 entrants, at gold-medal level. Read those two results together: the model is inside the top ten of the strongest human field available, and the top of that field was still ahead in the track that rewards ten hours of judgment.

ARC Prize caps a single semi-private evaluation run at $10,000 and prints cost per task next to accuracy. That is not administrative tidiness. On a benchmark where more compute per task buys a harder search, accuracy without cost is not a result, because any score can be bought by anyone willing to pay enough per puzzle. The cap turns the benchmark into a question with two axes, and it is the clearest example here of a benchmark author designing against the way their own measure would otherwise be gamed.

State of the art asserts exactly one thing: the highest published score on a named benchmark, on a named split, under a named protocol, as of a named date. Four qualifiers. A headline keeps the model name and drops all of them, and the sentence still reads fine, which is why two “state of the art on coding” claims from the same week routinely turn out to be about different task sets measured different ways. Attach the four and most disagreements about which model is best dissolve into which cell somebody was quoting.

For a decade the nearest thing to a canonical registry was Papers with Code: 9,327 benchmark leaderboards and 79,817 links between papers and their implementations, free to use, the place you went to check whether a claimed record was actually a record. Meta shut it down in late July 2025 without notice. The data is frozen on GitHub and HuggingFace, the domain redirects to a trending-papers page, and nothing has replaced it. So when somebody says a model is state of the art, there is now no shared place to look it up, and the claim rests on whoever is repeating it.

A benchmark is useful while it has headroom. MMLU launched in 2020 with the best models near 43% and now sits above 90% for everything at the frontier, which is saturation: it can no longer tell two good models apart. Humanity’s Last Exam was built as the reply, 2,500 questions from nearly a thousand experts across more than 500 institutions in 50 countries. At launch in early 2025 the best score was 2.7%. A year later it is around a quarter. The evals tutorial covers what saturation does to your own test suite; here the point is narrower. Check headroom before you read a rank.

Draw the top six rows of a 500-task board with the error bar from chapter 02 and they form one blur. First place is 4.2 points above sixth, every interval overlaps its neighbours, and the ordering would shuffle if everyone re-ran tomorrow. The arena has the same shape for a different reason: intervals there are set by vote volume, and the thinly-voted rows carry ±30 rating points. Rank is the least informative column on either page, and it is the one every summary quotes.

Two responses to the ceiling problem. Aggregate indices, like Epoch AI’s capabilities index, combine many benchmarks into one versioned scale, so no single saturated exam decides the ordering. And continuous axes measure something with no maximum: METR’s 50% time horizon is the human-expert task length a model completes half the time, measured on over a hundred software, machine-learning and cybersecurity tasks against timed human baselines. It has doubled roughly every seven months since 2019, faster since 2024, and METR states that measurements above 16 hours are unreliable with the current task suite. A duration cannot reach 100%, so this metric does not expire the way a percentage does. It just gets harder to measure.