# What is MT-Bench? The multi-turn LLM judge benchmark explained

> MT-Bench is a set of 80 two-turn questions that GPT-4 grades from 1 to 10. What it tests, how scoring works, and where it falls short.

Source: https://www.honeyhive.ai/learn/mt-bench (updated 2026-11-10)

MT-Bench is a benchmark of 80 multi-turn questions that a strong language model, usually GPT-4, grades on a scale of 1 to 10. Zheng and colleagues at LMSYS introduced it in a June 2023 [paper](https://arxiv.org/abs/2306.05685), later accepted to the NeurIPS 2023 Datasets and Benchmarks Track, together with the idea that an LLM can stand in for human raters. That idea is now called [LLM-as-a-judge](/learn/llm-as-a-judge).

## What MT-Bench contains

The question set has 10 questions in each of 8 categories, according to the [paper](https://arxiv.org/html/2306.05685). Every question has a follow-up, so each one tests whether the model keeps track of the conversation.

| Category | What it probes |
|---|---|
| Writing | Open-ended composition |
| Roleplay | Staying in a persona |
| Extraction | Pulling facts out of text |
| Reasoning | Logic problems |
| Math | Calculation and word problems |
| Coding | Writing and fixing code |
| STEM knowledge | Science and engineering questions |
| Humanities and social science knowledge | Questions from those fields |

The LMSYS [announcement](https://lmsys.org/blog/2023-06-22-leaderboard/) of June 2023 describes it as "a challenging multi-turn question set designed to evaluate the conversational and instruction-following ability of models."

## How the judge grades

The paper describes three judging modes:

- Pairwise comparison: the judge sees two answers and picks a winner or a tie.
- Single-answer grading: the judge scores one answer.
- Reference-guided grading: the judge also gets a reference answer, which matters for math and reasoning.

The LMSYS leaderboard [used](https://lmsys.org/blog/2023-06-22-leaderboard/) single-answer grading by GPT-4. In the FastChat [code](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/README.md) that implements the benchmark, the judge gives each turn a score out of 10 and the scores are averaged across turns. The same README lists a pairwise mode against a baseline model (gpt-3.5-turbo by default) and an all-pairs mode.

## How well the judge agrees with people

The authors collected about 3,000 votes from 58 expert labelers on MT-Bench answers. The table shows agreement on first-turn questions, from the [paper](https://arxiv.org/html/2306.05685). The first column counts ties and position-inconsistent votes as ties, where random agreement would be 33%. The second uses only non-tie votes, where random is 50%.

| Pair | With ties | Without ties |
|---|---|---|
| GPT-4 pairwise and human | 66% | 85% |
| GPT-4 single-answer and human | 60% | 85% |
| Human and human | 63% | 81% |

On the crowdsourced Chatbot Arena votes, the paper [reports](https://arxiv.org/html/2306.05685) 87% agreement for GPT-4 pairwise judging on non-tie votes. The authors [conclude](https://arxiv.org/abs/2306.05685) that strong judges reach over 80% agreement, comparable to agreement between humans. The paper released the 80 questions, 3,000 expert votes and 30,000 conversations with human preferences.

## Known problems

The same paper [lists](https://arxiv.org/html/2306.05685) the judge's weaknesses:

- Position bias. GPT-4 gave consistent verdicts when answer order was swapped in 65.0% of cases, and favored the first answer in 30.0%. Claude-v1 was consistent in 23.8%.
- Verbosity bias. On a test that padded an answer with a repeated list, GPT-4 was fooled in 8.7% of cases, while Claude-v1 and GPT-3.5 failed in 91.3%.
- Self-enhancement. GPT-4 favored itself with a 10% higher win rate and Claude-v1 with a 25% higher one, though the authors say limited data keeps them from settling whether the bias exists.
- Math and reasoning. On 10 math questions, GPT-4 as judge failed in 14 of 20 judgments with the default prompt, 6 with chain-of-thought and 3 with a reference answer.

[Limits and biases of LLM-as-a-judge](/learn/llm-judge-limitations) covers these and later research in more detail.

MT-Bench has limits as a benchmark too. It is a static set of 80 questions. In April 2024 the LMSYS team [compared](https://lmsys.org/blog/2024-04-19-arena-hard/) it with their Arena-Hard-Auto benchmark and measured its separability, the share of model pairs it can tell apart with confidence, at 22.6%, against 87.4% for Arena-Hard-Auto. The comparison comes from the Arena-Hard authors themselves, who also note that their newer benchmark does not evaluate multi-turn conversations, which is what MT-Bench was built for.

Other teams have extended the multi-turn idea. [MT-Bench-101](https://arxiv.org/abs/2402.14762), an ACL 2024 paper, has 1,388 dialogues and 4,208 turns across 13 tasks.

## Running it

The FastChat repository has the code and the questions. The README describes three steps: generate answers with `gen_model_answer.py`, generate judgments with `gen_judgment.py` (which needs an OpenAI API key for the default GPT-4 judge), and print scores with `show_result.py`. You can reuse the idea in your own evals. Write the criteria into a judge prompt, grade with a reference answer where one exists, swap answer order for pairwise checks, and compare the judge with human labels on a sample. [LLM evaluators](https://docs.honeyhive.ai/v2/evaluators/llm) in HoneyHive take criteria written as a prompt and return a boolean, numeric or string score, on traces in production or in offline experiments. See [LLM evaluation](/learn/llm-evaluation) for where a judge fits among other metrics, and [judge versus human evaluation](/learn/llm-judge-vs-human-evaluation) for when to use a person.

## FAQ

### What is MT-Bench?

A benchmark of 80 multi-turn questions across 8 categories, graded by a strong LLM judge such as GPT-4. It was introduced in the June 2023 [paper](https://arxiv.org/abs/2306.05685) that proposed LLM-as-a-judge.

### How is MT-Bench scored?

In the FastChat code, the default judge is GPT-4 and single-answer grading is the recommended mode. The judge [scores each turn](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/README.md) out of 10 and the scores are averaged across turns.

### Is MT-Bench the same as Chatbot Arena?

No. MT-Bench is a fixed set of questions graded by a model. Chatbot Arena is a platform where people vote between anonymous answers to prompts they write. The [paper](https://arxiv.org/abs/2306.05685) introduces both. See [LMArena](/learn/lmarena).

### How accurate is the MT-Bench judge?

On non-tie votes GPT-4 [agreed](https://arxiv.org/html/2306.05685) with human experts 85% of the time on first-turn questions, against 81% between two humans. It still showed position, verbosity and self-enhancement bias and struggled with math.

### Is MT-Bench still useful?

As a quick, cheap check of chat quality, yes. As a way to rank close models, less so: the Arena-Hard authors [measured](https://lmsys.org/blog/2024-04-19-arena-hard/) its separability at 22.6%, against 87.4% for their newer benchmark.