# What is Agents' Last Exam (ALE)?

> Agents' Last Exam is a benchmark of long professional tasks across 55 sub-fields, graded by deterministic code. What it tests, how it scores and its limits.

Source: https://www.honeyhive.ai/learn/agents-last-exam (updated 2026-11-19)

Agents' Last Exam (ALE) is a benchmark that tests whether AI agents can complete real professional projects, not answer questions. The [paper](https://arxiv.org/abs/2606.05405), posted in June 2026 with more than 250 industry experts involved, covers over 1,000 tasks in non-physical industries. Each task gives the agent a virtual machine, input files and software, and a grader checks the deliverable. It sits with [GDPval](/learn/gdpval) and the other [agent benchmarks](/learn/agent-benchmarks) as a test of work output.

## Where it came from

A team of 379 authors led by Yiyou Sun, with Dawn Song as the last listed author, [submitted the paper](https://arxiv.org/abs/2606.05405) on June 3, 2026. The [leaderboard](https://snorkel.ai/leaderboard/agents-last-exam/) says ALE is co-led by UC Berkeley RDI and the RDI Foundation, with funding and contributions from Snorkel AI, and lists more than 40 academic institutions.

The authors [argue](https://arxiv.org/abs/2606.05405) that existing benchmarks do not measure real, economically valuable workflows, which they see as a main reason AI gains have not carried over into professional deployment.

## What the tasks look like

Domain experts [submit](https://arxiv.org/html/2606.05405) past projects that took them days or weeks of professional work. Each proposal specifies a description, input files, target software, the expected deliverable and an evaluation spec. A task then passes a first review, an engineering implementation with dry runs, and a peer-review check by an expert committee.

The [leaderboard](https://snorkel.ai/leaderboard/agents-last-exam/) lists example tasks:

- Parse an SEC 10-K filing into a structured financial schema.
- Price an option with Monte Carlo methods.
- Lay out a PCB in KiCad.
- Run a root-cause analysis of a Kubernetes payment API failure.
- Compute legal fees from billing records.

Scope follows the U.S. occupational taxonomy. The [paper](https://arxiv.org/abs/2606.05405) organizes tasks into 55 sub-fields grouped into 13 industry clusters, defined with reference to O*NET and SOC 2018, and leaves out physical work.

## How a task runs

Per the [paper](https://arxiv.org/html/2606.05405), each task runs in a remote virtual machine with four directories:

| Directory | Purpose |
|---|---|
| `input/` | Read-only assets |
| `software/` | The tools for the task |
| `output/` | The only place the agent can write |
| `reference/` | Hidden from the agent, used only for scoring |

The agent runs inside a harness such as Codex or Claude Code, so the result reflects the model and the [agent harness](/learn/agent-harness) together. The paper [tests](https://arxiv.org/html/2606.05405) many harnesses, including Codex, Claude Code, Cursor, Gemini CLI and its own ALE-Claw, and reports each harness with a model.

## How it is scored

Each task has an `evaluate()` function that [scores](https://arxiv.org/html/2606.05405) the output from 0 to 1. Most tasks use a gate-and-score pattern: a binary precondition must pass before a continuous quality metric counts, and failing the gate forces the task score to 0. The authors avoid LLM judges where possible, and where one is needed it answers narrow, evidence-anchored yes/no probes. GDPval's experimental automated grader, by contrast, is [based on](https://arxiv.org/html/2510.04374) GPT-5-high, an [LLM-as-a-judge](/learn/llm-as-a-judge) approach.

The [leaderboard](https://snorkel.ai/leaderboard/agents-last-exam/) reports two numbers:

| Metric | Meaning |
|---|---|
| Pass rate | Share of tasks the agent fully completed (strict success) |
| Score | Average graded outcome across all tasks, including partial credit |

Do not mix them up: a high score with a low pass rate means an agent gets most of the way on many tasks and finishes few.

## Difficulty tiers

The [paper](https://arxiv.org/html/2606.05405) splits its evaluation set into three tiers:

| Tier | Tasks | Description |
|---|---|---|
| Near-Term | 67 | Workflows current frontier agents can partially complete |
| Full-Spectrum | 55 | At least one task in each of the 55 subdomains |
| Last-Exam | 38 | The hardest workflows, where most agents achieve a 0% pass rate |

In the paper's [results](https://arxiv.org/html/2606.05405), the best overall pass rate was 24.0%, from Codex with GPT-5.5, and that configuration scored 0.0% on the Last-Exam tier.

## Versions and scores

ALE is a living benchmark. The [leaderboard](https://snorkel.ai/leaderboard/agents-last-exam/) shows v1.1, released October 26, 2026, with 122 public tasks across 55 sub-industries, and marks v1.0 as legacy. About every six months a new public subset with fresh instances is released, which limits leakage. The paper [notes](https://arxiv.org/html/2606.05405) that only 150 of 1,490 instances were public at the time of the paper, so outside evaluators see a small slice.

On the v1.1 [leaderboard](https://snorkel.ai/leaderboard/agents-last-exam/) on November 19, 2026, GPT-6.2 Astra with the Codex harness was at the top with a 47.5% pass rate and a 78.3% score, and the leaderboard notes that even the top result leaves more than half of the tasks unsolved. Treat any figure as a snapshot and check the leaderboard.

## Limits worth knowing

- **Harness and model are bundled.** Results depend on the harness, effort setting and model. The [leaderboard](https://snorkel.ai/leaderboard/agents-last-exam/) says some harnesses were tested with many model variants and others with one or two, so comparisons are not evenly sampled.
- **Excluded tasks.** The [leaderboard](https://snorkel.ai/leaderboard/agents-last-exam/) says one entry, [Fable 5.2](/learn/claude-fable-5-2) with Claude Code, excludes 9 tasks that were downgraded, so its numbers are averaged over fewer tasks.
- **Small public slice.** Only part of the corpus is [public](https://arxiv.org/html/2606.05405), and the public subset [rotates](https://snorkel.ai/leaderboard/agents-last-exam/).
- **Cost and time.** The paper [reports](https://arxiv.org/html/2606.05405) that a run costs $3 to $10 on average and takes tens of minutes to hours.
- **Tooling gaps.** The paper [says](https://arxiv.org/html/2606.05405) agents underuse graphical tools and often write Bash scripts instead of using GUI software.
- **Not your workflow.** A strong ALE result does not show an agent will do well on your files, tools and review standards.

## Using ALE with your own evals

ALE tells you which model and harness pair is ahead on broad professional work. To choose for your application, build a [golden dataset](/learn/golden-dataset) from your own cases and follow the [LLM evaluation](/learn/llm-evaluation) approach. ALE's design is worth copying: a sandboxed environment, a hidden reference and a deterministic grader keep scoring repeatable. For benchmarks that test narrower agent skills, see [SWE-bench](/learn/swe-bench), [Terminal-Bench](/learn/terminal-bench) and [tau-bench](/learn/tau-bench).

## FAQ

### What does ALE stand for?

Agents' Last Exam. The [paper](https://arxiv.org/abs/2606.05405) describes it as a benchmark of long-horizon, economically valuable professional workflows with verifiable outcomes.

### Who built Agents' Last Exam?

The [leaderboard page](https://snorkel.ai/leaderboard/agents-last-exam/) says it is co-led by UC Berkeley RDI and the RDI Foundation, with funding and contributions from Snorkel AI through its Open Benchmarks Grants program.

### Does ALE use an LLM as the judge?

No, by design. The [leaderboard](https://snorkel.ai/leaderboard/agents-last-exam/) says it uses hidden references plus deterministic graders. The paper says that where a judge is unavoidable it is limited to narrow yes/no probes.

### How is ALE different from Humanity's Last Exam?

[Humanity's Last Exam](/learn/humanitys-last-exam) asks closed-ended academic questions. ALE gives an agent a multi-hour professional task in a virtual machine and grades the files it produces.
