# What is ARC-AGI? The benchmark, its three versions and how to read a score

> ARC-AGI is François Chollet's benchmark of grid puzzles that tests how efficiently AI learns new rules. It has three versions, and the third is interactive.

Source: https://www.honeyhive.ai/learn/arc-agi (updated 2026-11-10)

ARC-AGI is a benchmark of grid puzzles created by François Chollet. Each task shows a few example pairs of input and output grids, and the solver has to work out the rule and apply it to a new input. Chollet [introduced it](https://arcprize.org/arc-agi) in 2019 to measure fluid intelligence, and it now exists in three versions run by ARC Prize.

## Why it exists

In "On the Measure of Intelligence," Chollet [argued](https://arxiv.org/abs/1911.01547) that skill at a task does not measure intelligence, because prior knowledge and training data can inflate performance and hide how well a system generalizes. He proposed measuring skill-acquisition efficiency instead. ARC is [built on](https://arxiv.org/abs/1911.01547) "an explicit set of priors designed to be as close as possible to innate human priors," so it does not reward acquired knowledge.

That is the main difference from a knowledge or coding benchmark. A model cannot get far by having seen the subject before, because every task is a small new rule. For benchmarks that test other skills, see [agent benchmarks](/learn/agent-benchmarks) and [SWE-bench](/learn/swe-bench).

## The three versions

| Version | Format | Task sets |
|---|---|---|
| [ARC-AGI-1](https://arcprize.org/arc-agi/1) | Static grid puzzles, usually about three example pairs per task | Training 400, public eval 400, semi-private eval 100, private eval 100 |
| [ARC-AGI-2](https://arcprize.org/arc-agi/2) | Same grid format, harder tasks | Training 1,000, public eval 120, semi-private eval 120, private eval 120 |
| [ARC-AGI-3](https://arcprize.org/arc-agi/3) | Interactive environments where an agent chooses actions | See the ARC-AGI-3 docs |

ARC-AGI-2's authors [kept the input and output format](https://arxiv.org/abs/2505.11831) of the original so existing research stays compatible, and added a newly curated, expanded set of tasks. ARC Prize says it [removed tasks](https://arcprize.org/arc-agi/2) that brute-force search could solve, including all the tasks solved in the original 2020 Kaggle contest. Its new tasks target symbolic interpretation, compositional reasoning and contextual rule application.

ARC-AGI-3 is [described](https://docs.arcprize.org/) as an interactive reasoning benchmark that measures how an agent generalizes in novel, unseen environments. The agent acts by calling a step function with an action and gets a scorecard at the end. ARC Prize says a 100% score [means](https://arcprize.org/arc-agi/3) agents can beat every game as efficiently as humans, and that the environments are 100% human-solvable.

## What a task looks like

In the [ARC-AGI-1 repository](https://github.com/fchollet/ARC-AGI), each task is a JSON file with a `train` list (typically 3 input and output pairs) and a `test` list (typically 1 pair). A grid is a list of lists of integers from 0 to 9, each standing for a color, and can be anywhere from 1x1 to 30x30. The test-taker gets 3 trials per test input, and only exact solutions count: every cell has to match, and the output dimensions have to be right.

## Public, semi-private and private sets

The split matters when you read a result. ARC Prize [describes](https://arcprize.org/arc-agi/2) the ARC-AGI-2 sets this way:

- The public training and evaluation sets are open, so a model may have seen them.
- The semi-private set is not public, but limited third parties may have seen it, for example through API calls.
- The private set is not public and is not exposed to third parties.

ARC Prize says the evaluation sets are statistically similar and that scores should be comparable within under 1 percentage point, assuming no overfitting. When a score is quoted, check which set produced it.

## Human baseline

For ARC-AGI-2, ARC Prize [ran a live study](https://arcprize.org/arc-agi/2) in San Diego in early 2025 with over 400 members of the public. It reports that every task was solved by at least 2 humans, many by more, "in under 2 attempts."

## How to read an ARC-AGI score

A headline number needs a few details before you can compare it with another:

1. Which version (1, 2 or 3), since each has its own tasks and format.
2. Which set: public eval, semi-private or private. Only the private set is kept from third parties entirely.
3. How many attempts the harness allowed per test input.
4. What each task cost. ARC Prize [says](https://arcprize.org/arc-agi/2) all ARC-AGI-2 reporting includes an efficiency metric, starting with cost.
5. Whether the number comes from the [verified or the community leaderboard](https://arcprize.org/leaderboard), which are listed separately.

ARC Prize's leaderboard plots cost per task against performance, and has a tab for each benchmark. Use it for current numbers rather than a copy on another site. In November 2026 the page also says ARC Prize 2026 has finished and that results will be announced on December 4, 2026.

## Limits

ARC-AGI measures one thing, rule induction on small puzzles. It says nothing about tool use, long-horizon work or the quality of an answer in a production app. Treat it as one signal alongside the task-level checks in [LLM evaluation](/learn/llm-evaluation).

## Tracking a puzzle benchmark yourself

If you run ARC-style tasks against your own models, fix the task set, the prompt and the number of attempts, and keep the exact-match rule. Because grading is a cell-by-cell comparison, a deterministic check is enough. HoneyHive [Python evaluators](https://docs.honeyhive.ai/v2/evaluators/python.md) can hold that kind of programmatic check, and [offline experiments](https://docs.honeyhive.ai/v2/evaluation/introduction.md) run it over a dataset with `evaluate()`.

## FAQ

### What does ARC-AGI stand for?

Abstraction and Reasoning Corpus for Artificial General Intelligence. François Chollet [introduced it](https://arcprize.org/arc-agi) in his 2019 paper "On the Measure of Intelligence".

### Who created ARC-AGI?

François Chollet, in a [paper submitted to arXiv in November 2019](https://arxiv.org/abs/1911.01547). The ARC Prize organization runs the later versions and the leaderboards.

### How many tasks are in ARC-AGI-1 and ARC-AGI-2?

ARC-AGI-1 has [400 training and 400 public evaluation tasks](https://arcprize.org/arc-agi/1), plus 100 semi-private and 100 private evaluation tasks. ARC-AGI-2 has a [training set of 1,000 tasks](https://arcprize.org/arc-agi/2) and 120 tasks in each of its public, semi-private and private evaluation sets.

### What does a model have to do on an ARC-AGI task?

Infer a rule from a few example input and output grids, then produce the output grid for a new input. In ARC-AGI-1 the [grid must match exactly](https://github.com/fchollet/ARC-AGI), cell for cell, with dimensions included.

### Why does ARC-AGI report cost?

ARC Prize [says](https://arcprize.org/arc-agi/2) cost is the most directly comparable efficiency measure between humans and AI, and that "Intelligence is about finding the solution efficiently, not exhaustively."
