# AIME benchmark for LLMs: what it measures and how to read scores

> AIME is a US math contest whose integer-answer problems, 30 a year through 2026, test LLM reasoning. Scores depend on setup and contamination.

Source: https://www.honeyhive.ai/learn/aime-benchmark (updated 2026-11-09)

The AIME benchmark scores a language model on problems from the American Invitational Mathematics Examination, a US high school contest. Each problem has an integer answer from 000 to 999, so grading is an exact match on the final answer. The test set is small: the [AIME 2025 dataset](https://huggingface.co/datasets/opencompass/AIME2025) holds 30 problems, and the [AIME 2024 dataset](https://huggingface.co/datasets/HuggingFaceH4/aime_2024) holds 30 as well.

## Where the problems come from

The real exam has [15 questions of roughly increasing difficulty and a 3-hour limit](https://en.wikipedia.org/wiki/American_Invitational_Mathematics_Examination), with one point per correct answer and no partial credit. From 2000 to 2026 it was [offered twice a year](https://en.wikipedia.org/wiki/American_Invitational_Mathematics_Examination), as AIME I and AIME II, and the LLM datasets combine both, which is why they have 30 problems. The 2027 AIME moves to testing centers with [no alternate AIME II date](https://en.wikipedia.org/wiki/American_Invitational_Mathematics_Examination), so a 2027 set would have 15 problems.

| Dataset | Problems | Fields | Source |
|---|---|---|---|
| AIME 2025 (I and II) | 30 | question, answer | [Hugging Face](https://huggingface.co/datasets/opencompass/AIME2025) |
| AIME 2024 (I and II) | 30 | id, problem, solution, answer, url, year | [Hugging Face](https://huggingface.co/datasets/HuggingFaceH4/aime_2024) |

## Why it works as an LLM benchmark

- **Verifiable answers.** An integer from 000 to 999 can be checked by exact match, with no judge model needed. For open-ended outputs where that is not possible, see [LLM-as-a-judge](/learn/llm-as-a-judge).
- **Competition math.** The problems come from a contest, so they are a common stand-in for reasoning tests. Prompting models to show steps is covered in [chain-of-thought prompting](/learn/chain-of-thought-prompting).
- **Fresh problems every year.** New problems come out each year, which helps against memorization, as the next section explains.

## Limits to know before trusting a score

**Contamination.** The MathArena authors [wrote](https://arxiv.org/abs/2505.23281) in May 2025 that common math benchmarks such as AIME 2024 are "widely available online," which makes it hard to separate reasoning from memorization. They report "strong signs of contamination in AIME 2024." MathArena [tests models on new problems](https://matharena.ai/) so that they measure reasoning on questions the models have not seen during training.

**Small sample.** With 30 problems, one problem is 3.3 percentage points of accuracy. That is why leaderboards repeat runs: MathArena [runs each model four times on each problem](https://matharena.ai/) and reports the average score (as of November 2026).

**Setup changes the number.** Run count, prompt, final-answer extraction and the year of problems are choices the evaluator makes, so two headline scores are comparable only when these match. Record them next to every score, the same discipline as for [BLEU](/learn/bleu-score).

**Not the same as real tasks.** AIME checks competition math. It says little about tool use or multi-step work in production, which [agent benchmarks](/learn/agent-benchmarks) cover.

## Reading an AIME result

When you see an AIME number in a model announcement, check four things:

1. Which year, and whether the problems came out after the model's training data ended.
2. How many runs per problem, and whether the score is an average.
3. How the final answer was pulled from the output.
4. Whether the prompt and settings match the ones you would use.

For the wider picture of metrics and benchmarks, see [LLM evaluation](/learn/llm-evaluation).

## Running AIME-style checks on your own models

If you track a math benchmark across model or prompt changes, keep the problem set, the prompt and the run count fixed. Because answers are integers, an exact-match check is enough. HoneyHive [Python evaluators](https://docs.honeyhive.ai/v2/evaluators/python.md) can hold that kind of deterministic check, and [client-side evaluators](https://docs.honeyhive.ai/v2/evaluators/client_side.md) run scoring in your own code during experiments.

## FAQ

### What does AIME stand for?

American Invitational Mathematics Examination, a US high school math contest. [Wikipedia](https://en.wikipedia.org/wiki/American_Invitational_Mathematics_Examination) describes 15 questions of roughly increasing difficulty, answered in 3 hours.

### How many problems are in AIME 2025 for LLMs?

30. The [AIME2025 dataset](https://huggingface.co/datasets/opencompass/AIME2025) on Hugging Face has two subsets, AIME2025-I and AIME2025-II, with 15 problems each.

### Why are AIME answers easy to grade?

Each answer is an integer from 000 to 999, so a model's final answer can be checked by exact match, with [no partial credit](https://en.wikipedia.org/wiki/American_Invitational_Mathematics_Examination) on the real exam.

### Is AIME 2024 contaminated?

The MathArena authors [report](https://arxiv.org/abs/2505.23281) strong signs of contamination in AIME 2024. Treat old-year scores with care and prefer problems released after a model's training data ends.

### Can I compare AIME scores across model announcements?

Only when the setup matches. The number of runs per problem, how the final answer is extracted and which year's problems are used all change the result. MathArena [runs each model four times on each problem](https://matharena.ai/), for example.
