What is ARC-AGI? The benchmark, its three versions and how to read a score

Evaluate agents
HoneyHive Engineering

ARC-AGI is François Chollet's benchmark of grid puzzles that tests how efficiently AI learns new rules. It has three versions, and the third is interactive.

ARC-AGI is a benchmark of grid puzzles created by François Chollet. Each task shows a few example pairs of input and output grids, and the solver has to work out the rule and apply it to a new input. Chollet introduced it in 2019 to measure fluid intelligence, and it now exists in three versions run by ARC Prize.

Why it exists

In “On the Measure of Intelligence,” Chollet argued that skill at a task does not measure intelligence, because prior knowledge and training data can inflate performance and hide how well a system generalizes. He proposed measuring skill-acquisition efficiency instead. ARC is built on “an explicit set of priors designed to be as close as possible to innate human priors,” so it does not reward acquired knowledge.

That is the main difference from a knowledge or coding benchmark. A model cannot get far by having seen the subject before, because every task is a small new rule. For benchmarks that test other skills, see agent benchmarks and SWE-bench.

The three versions

Version Format Task sets
ARC-AGI-1 Static grid puzzles, usually about three example pairs per task Training 400, public eval 400, semi-private eval 100, private eval 100
ARC-AGI-2 Same grid format, harder tasks Training 1,000, public eval 120, semi-private eval 120, private eval 120
ARC-AGI-3 Interactive environments where an agent chooses actions See the ARC-AGI-3 docs

ARC-AGI-2’s authors kept the input and output format of the original so existing research stays compatible, and added a newly curated, expanded set of tasks. ARC Prize says it removed tasks that brute-force search could solve, including all the tasks solved in the original 2020 Kaggle contest. Its new tasks target symbolic interpretation, compositional reasoning and contextual rule application.

ARC-AGI-3 is described as an interactive reasoning benchmark that measures how an agent generalizes in novel, unseen environments. The agent acts by calling a step function with an action and gets a scorecard at the end. ARC Prize says a 100% score means agents can beat every game as efficiently as humans, and that the environments are 100% human-solvable.

What a task looks like

In the ARC-AGI-1 repository, each task is a JSON file with a train list (typically 3 input and output pairs) and a test list (typically 1 pair). A grid is a list of lists of integers from 0 to 9, each standing for a color, and can be anywhere from 1x1 to 30x30. The test-taker gets 3 trials per test input, and only exact solutions count: every cell has to match, and the output dimensions have to be right.

Public, semi-private and private sets

The split matters when you read a result. ARC Prize describes the ARC-AGI-2 sets this way:

  • The public training and evaluation sets are open, so a model may have seen them.
  • The semi-private set is not public, but limited third parties may have seen it, for example through API calls.
  • The private set is not public and is not exposed to third parties.

ARC Prize says the evaluation sets are statistically similar and that scores should be comparable within under 1 percentage point, assuming no overfitting. When a score is quoted, check which set produced it.

Human baseline

For ARC-AGI-2, ARC Prize ran a live study in San Diego in early 2025 with over 400 members of the public. It reports that every task was solved by at least 2 humans, many by more, “in under 2 attempts.”

How to read an ARC-AGI score

A headline number needs a few details before you can compare it with another:

  1. Which version (1, 2 or 3), since each has its own tasks and format.
  2. Which set: public eval, semi-private or private. Only the private set is kept from third parties entirely.
  3. How many attempts the harness allowed per test input.
  4. What each task cost. ARC Prize says all ARC-AGI-2 reporting includes an efficiency metric, starting with cost.
  5. Whether the number comes from the verified or the community leaderboard, which are listed separately.

ARC Prize’s leaderboard plots cost per task against performance, and has a tab for each benchmark. Use it for current numbers rather than a copy on another site. In November 2026 the page also says ARC Prize 2026 has finished and that results will be announced on December 4, 2026.

Limits

ARC-AGI measures one thing, rule induction on small puzzles. It says nothing about tool use, long-horizon work or the quality of an answer in a production app. Treat it as one signal alongside the task-level checks in LLM evaluation.

Tracking a puzzle benchmark yourself

If you run ARC-style tasks against your own models, fix the task set, the prompt and the number of attempts, and keep the exact-match rule. Because grading is a cell-by-cell comparison, a deterministic check is enough. HoneyHive Python evaluators can hold that kind of programmatic check, and offline experiments run it over a dataset with evaluate().

FAQ

What does ARC-AGI stand for?

Abstraction and Reasoning Corpus for Artificial General Intelligence. François Chollet introduced it in his 2019 paper "On the Measure of Intelligence".

Who created ARC-AGI?

François Chollet, in a paper submitted to arXiv in November 2019. The ARC Prize organization runs the later versions and the leaderboards.

How many tasks are in ARC-AGI-1 and ARC-AGI-2?

ARC-AGI-1 has 400 training and 400 public evaluation tasks, plus 100 semi-private and 100 private evaluation tasks. ARC-AGI-2 has a training set of 1,000 tasks and 120 tasks in each of its public, semi-private and private evaluation sets.

What does a model have to do on an ARC-AGI task?

Infer a rule from a few example input and output grids, then produce the output grid for a new input. In ARC-AGI-1 the grid must match exactly, cell for cell, with dimensions included.

Why does ARC-AGI report cost?

ARC Prize says cost is the most directly comparable efficiency measure between humans and AI, and that "Intelligence is about finding the solution efficiently, not exhaustively."

Share