# What is MMLU (and MMLU-Pro)? The benchmark explained

> MMLU is a 57-subject multiple-choice test of model knowledge; MMLU-Pro raises it to ten options and harder reasoning questions. What each measures and misses.

Source: https://www.honeyhive.ai/learn/mmlu (updated 2026-11-12)

MMLU is a multiple-choice benchmark that tests how much a language model knows across [57 subjects](https://arxiv.org/abs/2009.03300), from elementary mathematics to law. Dan Hendrycks and colleagues [published](https://arxiv.org/abs/2009.03300) it in September 2020. MMLU-Pro is a harder follow-up that [expands](https://arxiv.org/abs/2406.01574) each question from four answer options to ten and leans on reasoning. Both report accuracy, the share of questions answered correctly.

They sit in the benchmark family next to [SimpleQA](/learn/simpleqa) and the [AIME benchmark](/learn/aime-benchmark), and they are the usual general-knowledge column in model comparisons. For how benchmarks fit with testing your own application, see [LLM evaluation](/learn/llm-evaluation).

## What MMLU contains

The [authors collected](https://ar5iv.labs.arxiv.org/html/2009.03300) 15,908 questions, which graduate and undergraduate students gathered from free online sources such as practice exams for the GRE and USMLE, undergraduate course questions and questions for Oxford University Press book readers. The paper splits them into a few-shot development set (285 examples in the [Hugging Face copy](https://huggingface.co/datasets/cais/mmlu), about 5 per subject), a validation set of 1,540 and a test set of 14,079.

Subjects fall into four groups: Humanities, Social Science, STEM and Other. Each is tagged Elementary, High School, College or Professional. Every question has [four options](https://huggingface.co/datasets/cais/mmlu).

The [paper](https://arxiv.org/abs/2009.03300) found that most models then performed near random chance, and that GPT-3 gained almost 20 percentage points over chance on average. It also found uneven performance across subjects and that models often did not know when they were wrong. On the human side, the [authors report](https://ar5iv.labs.arxiv.org/html/2009.03300) 34.5% for unspecialized crowd workers and estimate expert-level accuracy at 89.8%.

## How an MMLU score is produced

The [original prompt format](https://ar5iv.labs.arxiv.org/html/2009.03300) begins "The following are multiple choice questions (with answers) about [subject]", adds up to 5 worked examples, and ends with "Answer: ". The model assigns a probability to each of A, B, C and D, and the highest one is its answer. The [repository](https://github.com/hendrycks/test) has the evaluation code under an MIT license.

So a score depends on the setup: the number of examples, the prompt wording and how the answer is read out. A chat model scored by generating text and then parsing the letter is measured differently from one scored by option probabilities.

## What MMLU-Pro changes

The MMLU-Pro [paper](https://arxiv.org/abs/2406.01574) was submitted in June 2024 and accepted to the NeurIPS 2024 Datasets and Benchmarks track. The [paper](https://arxiv.org/html/2406.01574v6) describes the changes:

| Change | What the paper says |
|---|---|
| Size | 12,032 questions in 14 discipline subsets. |
| Sources | 6,810 questions from MMLU, 4,083 from the STEM Website, 598 from TheoremQA and 541 from SciBench. |
| Filtering | Eight open models answered MMLU, and questions that more than four answered correctly were dropped as too easy, removing 5,886. |
| Options | GPT-4-Turbo added six distractors to reach ten options. After expert review, 83% of questions have ten, with an average of 9.47. |
| Review | Experts checked answers and removed unsuitable questions, and a second pass used Gemini-1.5-Pro to flag possible false-negative options for human review. |

The authors [report](https://arxiv.org/abs/2406.01574) that accuracy falls 16% to 33% against MMLU, and that sensitivity to prompt wording across 24 prompt styles drops from about 4-5% to about 2%. They also found that chain-of-thought prompting beats direct answering on MMLU-Pro, the reverse of the original MMLU, which they attribute to its more reasoning-heavy questions.

That makes the setup matter more. The [paper](https://arxiv.org/html/2406.01574v6) evaluates most models with 5-shot [chain-of-thought](/learn/chain-of-thought-prompting) prompting and extracts answers with a regular expression. The [dataset card](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) says direct evaluation can drop substantially. The repository hosts the [24 prompt styles](https://github.com/TIGER-AI-Lab/MMLU-Pro) and scripts for any OpenAI-compatible API, and the [leaderboard](https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro) lives on Hugging Face.

## MMLU compared with MMLU-Pro

| | MMLU | MMLU-Pro |
|---|---|---|
| Introduced | [September 2020](https://arxiv.org/abs/2009.03300) | [June 2024](https://arxiv.org/abs/2406.01574) |
| Options per question | 4 | [Up to 10](https://arxiv.org/abs/2406.01574) |
| Subjects | [57 tasks](https://arxiv.org/abs/2009.03300) | [14 disciplines](https://arxiv.org/html/2406.01574v6) |
| Questions | [15,908](https://ar5iv.labs.arxiv.org/html/2009.03300) | [12,032](https://arxiv.org/html/2406.01574v6) |
| Usual prompting | [Few-shot, answer letter](https://ar5iv.labs.arxiv.org/html/2009.03300) | [5-shot chain of thought](https://arxiv.org/html/2406.01574v6) |

## Where the scores stop being useful

- **Label errors.** The MMLU-Redux authors [estimate](https://arxiv.org/abs/2406.04127) that 6.49% of MMLU questions contain errors, and that 57% of the Virology questions they analysed do. They released a [subset](https://arxiv.org/abs/2406.04127) of 5,700 questions re-annotated by hand.
- **Copies differ.** The Hugging Face [MMLU dataset](https://huggingface.co/datasets/cais/mmlu) lists 14,042 test examples, while the paper has 14,079, so check which copy and which harness a reported number used.
- **MMLU-Pro is still being corrected.** Its dataset card [lists](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) a fix to 15 medical answers in April 2025, a note on Health-category issues in October 2025 and a leading-space fix in answer options affecting STEM subsets in January 2026.
- **Multiple choice is not your product.** Picking a letter is far from writing a support reply or calling a tool. For tasks that act, look at [agent benchmarks](/learn/agent-benchmarks) and the [LLM evaluation metrics](/learn/llm-evaluation-metrics) page.
- **Scores move with the prompt.** The MMLU-Pro authors [measured](https://arxiv.org/abs/2406.01574) prompt sensitivity of about 2% even on the harder set, so a gap of a point or two between two models can come from the setup. Compare scores only when shots, prompt and extraction match.

## Using it in your own evals

Use MMLU-style scoring as a template for a private test set: fixed questions, one checkable answer, and the same prompt for every model you compare. A [golden dataset](/learn/golden-dataset) built from your own cases tells you more about your product than a public number. In HoneyHive you can [upload a dataset](https://docs.honeyhive.ai/v2/datasets/import.md), score outputs with a [Python evaluator](https://docs.honeyhive.ai/v2/evaluators/python.md) that checks the chosen letter, and compare runs as described in the [experiments guide](https://docs.honeyhive.ai/v2/evaluation/introduction.md).

## FAQ

### What does MMLU stand for?

Massive Multitask Language Understanding. The [paper](https://arxiv.org/abs/2009.03300) by Hendrycks and colleagues introduced it in September 2020 as a test over 57 tasks.

### How many questions are in MMLU?

The [authors collected](https://ar5iv.labs.arxiv.org/html/2009.03300) 15,908 questions, with a test set of 14,079. The Hugging Face [copy](https://huggingface.co/datasets/cais/mmlu) lists 14,042 test examples, so counts differ between copies.

### What is the difference between MMLU and MMLU-Pro?

MMLU-Pro [expands](https://arxiv.org/abs/2406.01574) the choices from four to ten, removes easy questions and adds more reasoning-heavy ones. Its authors report accuracy drops of 16% to 33% compared with MMLU.

### Is MMLU still a good benchmark?

It is a quick general-knowledge check, but its labels have problems. The MMLU-Redux authors [estimate](https://arxiv.org/abs/2406.04127) that 6.49% of questions contain errors, and it says little about your own task.

### Which prompting setup does MMLU-Pro use?

The paper [evaluates](https://arxiv.org/html/2406.01574v6) most models with 5-shot chain-of-thought prompting, and the dataset card [says](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) direct answering can score substantially lower.
