# What is the GAIA benchmark?

> GAIA is a benchmark of real-world assistant questions that need browsing, tools and reasoning, scored by exact match on three difficulty levels.

Source: https://www.honeyhive.ai/learn/gaia-benchmark (updated 2026-11-11)

GAIA is a benchmark of real-world questions for general AI assistants. Each question needs several abilities at once, such as web browsing, reading a file, using tools and reasoning, and each has one short answer that is easy to check. Mialon and colleagues from Meta, Hugging Face and AutoGPT introduced it in a [November 2023 paper](https://arxiv.org/abs/2311.12983) with 466 questions. It is one of the benchmarks covered in [agent benchmarks](/learn/agent-benchmarks).

The questions are conceptually simple for people. The paper [reports](https://arxiv.org/abs/2311.12983) that human respondents scored 92%, against 15% for GPT-4 equipped with plugins, the strongest setup tested in 2023.

## What a GAIA question looks like

The [paper](https://arxiv.org/html/2311.12983v1) describes text questions that sometimes come with a file such as an image or a spreadsheet. They cover everyday personal tasks, science and general knowledge. One Level 1 example asks for the actual enrollment count of a clinical trial listed on the NIH website. The answer is a single number.

Annotators [followed](https://arxiv.org/html/2311.12983v1) these rules when writing questions:

- Base the question on a source of truth such as Wikipedia, arXiv or GitHub, and combine sources for harder levels.
- The answer must not appear on the internet in plain text.
- The answer is a number or a few words.
- It does not change over time.
- It is unambiguous.

## The three levels

The [paper](https://arxiv.org/html/2311.12983v1) assigns levels loosely, by the number of steps and tools a question needs:

| Level | Roughly what it needs |
|---|---|
| 1 | No tools, or at most one tool, and no more than 5 steps |
| 2 | About 5 to 10 steps, and a combination of tools |
| 3 | Arbitrarily long action sequences and any number of tools, for a "near perfect general assistant" |

The authors say these are not hard rules. A question with fewer than 10 steps but complex web navigation might be Level 3. Human annotators [averaged](https://arxiv.org/html/2311.12983v1) 6.8 minutes per Level 1 question and 17.7 minutes per Level 3 question.

## How answers are scored

Each question has one correct answer: a string, a number, or a comma separated list. The [paper](https://arxiv.org/html/2311.12983v1) scores it by quasi exact match against the ground truth, with normalization tied to the answer type, and a system prompt tells the model which format to use. The authors publish the scoring function with the leaderboard.

Because only the final answer is graded, there is no judge model to configure, unlike in [LLM-as-a-judge](/learn/llm-as-a-judge) setups. The paper adds that this makes GAIA robust to randomness in token generation.

## How the questions were validated

After creation, each question was [answered](https://arxiv.org/html/2311.12983v1) by two new annotators to check it was not ambiguous. Of 623 newly written questions, 68% were valid, and the share fell with difficulty: 75% for Level 1, 68% for Level 2 and 47% for Level 3. On the valid questions, humans scored 94% at Level 1, 92% at Level 2 and 87% at Level 3.

## Data and leaderboard

The dataset sits on Hugging Face. The [dataset card](https://huggingface.co/datasets/gaia-benchmark/GAIA) says each level has a public validation set and a test set with private answers and metadata. The card says it has "more than 450" questions, while the paper counts 466, so check the card for the current split sizes.

The files are gated to stop bots scraping them. The card asks users not to reshare them outside a gated or private repository, or to publish the validation or test sets in a crawlable format. Some questions have attachments named in the `file_name` field. The [leaderboard](https://huggingface.co/spaces/gaia-benchmark/leaderboard) is a Hugging Face Space.

Third parties also run GAIA. Princeton's HAL [uses](https://hal.cs.princeton.edu/gaia) the 165-question public validation set. As of 11 November 2026 its top entry was a HAL Generalist Agent running Claude Sonnet 4.5, at 74.55% accuracy and a cost of $178.20. HAL also [notes](https://hal.cs.princeton.edu/gaia) that it paused adding new models, and that costs and token counts in some rows were overstated because of double counted spans. Compare scores only when the scaffold, model, split and date match.

## What GAIA does not tell you

- It does not grade the path. The paper [says](https://arxiv.org/html/2311.12983v1) different paths can reach the same answer and that it leaves model-based or human grading of plans for future work. Two agents with the same score can differ in cost, retries and failures. See [agent tracing](/learn/agent-tracing) for inspecting those.
- It is English only. The authors [note](https://arxiv.org/html/2311.12983v1) that many questions rely on English web pages, so it does not show how well an assistant serves non-English speakers.
- It is small on purpose and can decay. The authors [expect](https://arxiv.org/html/2311.12983v1) contamination of pre-training data, or sources disappearing from the web, to erode it over time.
- Early results are hard to reproduce. The paper [explains](https://arxiv.org/html/2311.12983v1) that the GPT-4 plugin results were collected by manually selecting plugins, and plugin capabilities change.
- It covers assistants answering questions, not acting in an environment. For tool-use conversations with policies see [tau-bench](/learn/tau-bench), for terminal tasks [Terminal-Bench](/learn/terminal-bench) and for software fixes [SWE-bench](/learn/swe-bench).

## Using the idea in your own evals

GAIA's design carries over to a private test set: questions with one checkable answer, a difficulty tag per question, and a format instruction in the prompt. Pair it with a check on the trace, since the final answer alone hides how an agent got there. HoneyHive's docs [explain](https://docs.honeyhive.ai/v2/evaluation/concepts.md) how experiments combine datasets, functions and evaluators, and [AI agent evaluation](/learn/ai-agent-evaluation) covers what to measure beyond the answer.

## FAQ

### What does GAIA stand for?

The [paper](https://arxiv.org/abs/2311.12983) calls it a benchmark for General AI Assistants. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun and Thomas Scialom submitted it on 21 November 2023.

### How many questions does GAIA have?

The paper [presents](https://arxiv.org/abs/2311.12983) 466 questions and withholds the answers to 300 for a leaderboard. The 166 with public answers are the developer set.

### How is GAIA scored?

By [quasi exact match](https://arxiv.org/html/2311.12983v1) between the final answer and the single ground truth, after normalization that depends on the answer type.

### Can I download the GAIA dataset?

Yes, after accepting its conditions. The [dataset card](https://huggingface.co/datasets/gaia-benchmark/GAIA) gates the files and asks users not to republish the validation or test sets in a crawlable format.

### Does GAIA grade the steps an agent takes?

No. The authors [write](https://arxiv.org/html/2311.12983v1) that GAIA does not evaluate the trace leading to the answer, only the final answer.