What is GPQA Diamond? The graduate-level science benchmark
GPQA Diamond is a 198-question subset of graduate-level biology, physics and chemistry questions that experts solve and skilled non-experts miss.
GPQA Diamond is the hardest 198-question subset of GPQA, a benchmark of graduate-level multiple-choice questions in biology, physics and chemistry. A harder, expert-written successor in spirit is Humanity’s Last Exam. Domain experts write the questions, and Diamond keeps only those that experts answered correctly and most skilled non-experts with web access got wrong. Model release posts often quote a GPQA Diamond score, and the details below decide how far that number can be trusted.
Rein and colleagues released GPQA in November 2023. They built it so that a model’s answers could be hard to check by search, and so that researchers could study how humans might supervise AI systems that know more than they do.
The three sets
The paper defines three nested sets. The accuracy figures for Main and Diamond are skewed by how the sets were selected, so they are not a clean estimate of human skill.
| Set | Questions | Expert accuracy | Non-expert accuracy |
|---|---|---|---|
| Extended | 546 | 64.8% | 34.1% |
| Main | 448 | 71.9% | 30.4% |
| Diamond | 198 | 81.3% | 22.1% |
Diamond keeps a question only when both expert validators answered it correctly and at most one third of non-experts did. The paper’s headline GPQA numbers, 65% for experts and 34% for non-experts, match the Extended set’s figures, not Diamond’s. The strongest GPT-4 baseline in the paper scored 38.8% on Diamond.
How the questions were made
Sixty-one Upwork contractors with a PhD or PhD enrolment wrote and validated the questions. Each question went through four steps:
- One expert wrote it.
- A first expert validator answered and commented on it.
- The writer revised it.
- A second expert validator answered the revised version.
Three non-experts from other fields then tried each question. They had unrestricted time and internet access but no LLM assistants, had to spend at least 15 minutes per question, and averaged 37 minutes. That is what “Google-proof” means here. The authors also note that the label is a design goal, not a guarantee.
How models are run on it
The GPQA repository includes baselines for closed-book, open-book retrieval and answer-only settings. Model reports typically prompt the model and extract a letter answer, and the details matter: Epoch AI uses the simple-evals prompt, which asks the model to end with “ANSWER: LETTER” and gives no credit if the format is missing. With four options, chance is 25%.
OpenAI’s simple-evals repository has a GPQA evaluation, but a July 2025 notice says the repo is no longer updated for new models or results and names only HealthBench, BrowseComp and SimpleQA as the reference implementations it will keep hosting.
Limits to keep in mind
- Small size. The paper lists small size as a limitation. With 198 items, one question is about half a percentage point, so small gaps between models are weak evidence.
- Possible bad questions. Greg Burnham of Epoch AI reviewed six of the seven questions that models almost never answered, and extrapolated to about 15 of 198 (about 8%) invalid, with a caveat about the small sample. He still judged that at least 90% were likely valid. Near the top of the range, part of the remaining gap may be the answer key.
- Multiple choice. Burnham argues that future versions should move away from multiple choice and test research-assistant skills.
- Human baselines are soft. The paper notes that its non-experts were highly skilled, so their accuracy is an upper bound, and that experts came from a contractor pool without enforced demographic representation.
- Format and harness effects. Scores shift with the prompt, the answer-extraction rule and whether refusals or API errors count as wrong. Epoch AI changed its expected answer format in February 2026, for example.
- Leakage. The data carries a canary string, and the dataset card asks users not to reveal examples online. Still, any public benchmark can leak into training data over time.
GPQA Diamond tests recall and reasoning on closed science questions. It says little about tool use, long tasks or agent behavior; for those, see agent benchmarks. For another benchmark with short, checkable answers, compare AIME and SimpleQA, and read LLM evaluation metrics for the general picture.
Building a private version
The design lesson is to keep a private set with a known answer key, validated by more than one person, and to treat the hardest questions as candidates for relabeling when every model fails them. See golden dataset for how to build one. HoneyHive’s experiments run a dataset through a function and score outputs with Python evaluators, which suits exact-match multiple-choice checks.
FAQ
What does GPQA stand for?
Graduate-Level Google-Proof Q&A. The paper introduces it as 448 multiple-choice questions written by domain experts in biology, physics and chemistry.
How many questions are in GPQA Diamond?
198. The paper keeps only questions that both expert validators answered correctly and that at most one third of non-experts answered correctly.
What is a random-guess score on GPQA Diamond?
25%, because each question has four options, as Epoch AI notes. Its harness gives no credit for a response that lacks the required answer format, so a score can even fall below 25%.
Is GPQA Diamond saturated?
Not fully, though some of the remaining headroom may be answer-key error. Epoch AI's Greg Burnham estimated in May 2025 that about 8% of Diamond questions may be invalid, from a small sample, while concluding the benchmark still had a bit more juice left.
Can I use GPQA questions in my own prompts or blog posts?
Not in plain text or images. The Hugging Face dataset card asks users to agree not to reveal examples online, to keep them out of training corpora.