What is Agents' Last Exam (ALE)?

Evaluate agents
HoneyHive Engineering

Agents' Last Exam is a benchmark of long professional tasks across 55 sub-fields, graded by deterministic code. What it tests, how it scores and its limits.

Agents’ Last Exam (ALE) is a benchmark that tests whether AI agents can complete real professional projects, not answer questions. The paper, posted in June 2026 with more than 250 industry experts involved, covers over 1,000 tasks in non-physical industries. Each task gives the agent a virtual machine, input files and software, and a grader checks the deliverable. It sits with GDPval and the other agent benchmarks as a test of work output.

Where it came from

A team of 379 authors led by Yiyou Sun, with Dawn Song as the last listed author, submitted the paper on June 3, 2026. The leaderboard says ALE is co-led by UC Berkeley RDI and the RDI Foundation, with funding and contributions from Snorkel AI, and lists more than 40 academic institutions.

The authors argue that existing benchmarks do not measure real, economically valuable workflows, which they see as a main reason AI gains have not carried over into professional deployment.

What the tasks look like

Domain experts submit past projects that took them days or weeks of professional work. Each proposal specifies a description, input files, target software, the expected deliverable and an evaluation spec. A task then passes a first review, an engineering implementation with dry runs, and a peer-review check by an expert committee.

The leaderboard lists example tasks:

  • Parse an SEC 10-K filing into a structured financial schema.
  • Price an option with Monte Carlo methods.
  • Lay out a PCB in KiCad.
  • Run a root-cause analysis of a Kubernetes payment API failure.
  • Compute legal fees from billing records.

Scope follows the U.S. occupational taxonomy. The paper organizes tasks into 55 sub-fields grouped into 13 industry clusters, defined with reference to O*NET and SOC 2018, and leaves out physical work.

How a task runs

Per the paper, each task runs in a remote virtual machine with four directories:

Directory Purpose
input/ Read-only assets
software/ The tools for the task
output/ The only place the agent can write
reference/ Hidden from the agent, used only for scoring

The agent runs inside a harness such as Codex or Claude Code, so the result reflects the model and the agent harness together. The paper tests many harnesses, including Codex, Claude Code, Cursor, Gemini CLI and its own ALE-Claw, and reports each harness with a model.

How it is scored

Each task has an evaluate() function that scores the output from 0 to 1. Most tasks use a gate-and-score pattern: a binary precondition must pass before a continuous quality metric counts, and failing the gate forces the task score to 0. The authors avoid LLM judges where possible, and where one is needed it answers narrow, evidence-anchored yes/no probes. GDPval’s experimental automated grader, by contrast, is based on GPT-5-high, an LLM-as-a-judge approach.

The leaderboard reports two numbers:

Metric Meaning
Pass rate Share of tasks the agent fully completed (strict success)
Score Average graded outcome across all tasks, including partial credit

Do not mix them up: a high score with a low pass rate means an agent gets most of the way on many tasks and finishes few.

Difficulty tiers

The paper splits its evaluation set into three tiers:

Tier Tasks Description
Near-Term 67 Workflows current frontier agents can partially complete
Full-Spectrum 55 At least one task in each of the 55 subdomains
Last-Exam 38 The hardest workflows, where most agents achieve a 0% pass rate

In the paper’s results, the best overall pass rate was 24.0%, from Codex with GPT-5.5, and that configuration scored 0.0% on the Last-Exam tier.

Versions and scores

ALE is a living benchmark. The leaderboard shows v1.1, released October 26, 2026, with 122 public tasks across 55 sub-industries, and marks v1.0 as legacy. About every six months a new public subset with fresh instances is released, which limits leakage. The paper notes that only 150 of 1,490 instances were public at the time of the paper, so outside evaluators see a small slice.

On the v1.1 leaderboard on November 19, 2026, GPT-6.2 Astra with the Codex harness was at the top with a 47.5% pass rate and a 78.3% score, and the leaderboard notes that even the top result leaves more than half of the tasks unsolved. Treat any figure as a snapshot and check the leaderboard.

Limits worth knowing

  • Harness and model are bundled. Results depend on the harness, effort setting and model. The leaderboard says some harnesses were tested with many model variants and others with one or two, so comparisons are not evenly sampled.
  • Excluded tasks. The leaderboard says one entry, Fable 5.2 with Claude Code, excludes 9 tasks that were downgraded, so its numbers are averaged over fewer tasks.
  • Small public slice. Only part of the corpus is public, and the public subset rotates.
  • Cost and time. The paper reports that a run costs $3 to $10 on average and takes tens of minutes to hours.
  • Tooling gaps. The paper says agents underuse graphical tools and often write Bash scripts instead of using GUI software.
  • Not your workflow. A strong ALE result does not show an agent will do well on your files, tools and review standards.

Using ALE with your own evals

ALE tells you which model and harness pair is ahead on broad professional work. To choose for your application, build a golden dataset from your own cases and follow the LLM evaluation approach. ALE’s design is worth copying: a sandboxed environment, a hidden reference and a deterministic grader keep scoring repeatable. For benchmarks that test narrower agent skills, see SWE-bench, Terminal-Bench and tau-bench.

FAQ

What does ALE stand for?

Agents' Last Exam. The paper describes it as a benchmark of long-horizon, economically valuable professional workflows with verifiable outcomes.

Who built Agents' Last Exam?

The leaderboard page says it is co-led by UC Berkeley RDI and the RDI Foundation, with funding and contributions from Snorkel AI through its Open Benchmarks Grants program.

Does ALE use an LLM as the judge?

No, by design. The leaderboard says it uses hidden references plus deterministic graders. The paper says that where a judge is unavoidable it is limited to narrow yes/no probes.

How is ALE different from Humanity's Last Exam?

Humanity's Last Exam asks closed-ended academic questions. ALE gives an agent a multi-hour professional task in a virtual machine and grades the files it produces.

Share