What is SimpleQA? Measuring factuality

Evaluate agents
HoneyHive Engineering

SimpleQA is OpenAI's benchmark of 4,326 short fact questions, graded correct, incorrect or not attempted. How it is built, scored and where it stops.

SimpleQA is a benchmark that tests whether a language model can answer short, fact-seeking questions, each with one indisputable answer. OpenAI researchers introduced it in a November 2024 paper, and it has 4,326 questions. Each model answer is graded correct, incorrect or not attempted, so the score rewards a model for saying nothing when it is unsure.

It measures one narrow slice of LLM hallucinations: wrong facts in short answers, with no documents supplied.

How the questions were built

The paper describes two steps. An AI trainer wrote question and answer pairs, then a second trainer answered each question independently, and a question stayed only if both answers matched.

Questions had to meet three criteria:

  • A single answer.
  • A reference answer that does not change over time.
  • Answerable as of December 31, 2023 (the paper’s cutoff).

The questions were also adversarially collected against GPT-4 responses, which is why they are hard for models of that generation.

A third trainer answered a sample and agreed with the reference 94.4% of the time. The authors estimate the benchmark’s error rate at around 3%.

How answers are graded

The paper’s grading table defines three labels:

Label Meaning
Correct The answer fully contains the reference answer without contradicting it.
Incorrect The answer contradicts the reference in any way, even if hedged.
Not attempted The reference answer is not fully given, and nothing contradicts it.

A grader model assigns the label. In the reference implementation, the caller supplies the grader model, which replies with the letter A, B or C, and a reply with no letter counts as not attempted. This makes SimpleQA an instance of LLM-as-a-judge, so the grader prompt and model are part of the setup.

Reading the scores

The paper reports two numbers beyond the share of correct answers. “Correct given attempted” is the percent correct out of only the questions the model tried. The F-score is the harmonic mean of overall correct and correct given attempted. The reference code reports the same quantities as accuracy_given_attempted and f1.

Because of this, two models with the same share correct can differ: the one that declines more of the questions it would miss has the higher correct given attempted. That is the behavior the authors want. Their stated aim is to test whether models “know what they know.”

The paper also tests stated confidence. It finds that models fall well below the line where confidence equals accuracy, meaning they consistently overstate their confidence, and that accuracy rises with how often the same answer is produced across samples.

What SimpleQA does not tell you

  • It covers short, single-answer facts. The authors say it only measures factuality in that constrained setting, and that the link to long-form factuality is an open question.
  • Questions are answerable as of the end of 2023, so it does not test knowledge of later events.
  • It tests what the model knows from training. It says nothing about whether an agent or a RAG pipeline stays faithful to retrieved documents.
  • It has label problems. Google DeepMind and Google Research authors built SimpleQA Verified, a 1,000-prompt version, to address noisy or incorrect labels, topical bias and redundant questions, using deduplication, topic balancing, source reconciliation and a revised grader prompt.
  • Reference code is frozen. The repository notes that, since July 2025, it is no longer updated for new models or results, though it still hosts the SimpleQA reference implementation.

When you compare numbers across reports, check which version (original or Verified), which grader and which prompt were used. Other public benchmarks are listed on agent benchmarks and in LLM evaluation metrics.

Using the idea in your own evals

SimpleQA’s design carries over to a private test set: one checkable answer per question, three grade labels, and a report of both accuracy and correct given attempted. HoneyHive supports Python evaluators for deterministic checks and LLM evaluators for model-graded labels.

FAQ

How many questions are in SimpleQA?

The paper says SimpleQA contains 4,326 short, fact-seeking questions.

Who made SimpleQA?

OpenAI researchers Jason Wei, Nguyen Karina, Hyung Won Chung and colleagues, in a paper submitted on 7 November 2024.

What is the F-score in SimpleQA?

The paper defines it as the harmonic mean of overall correct and correct given attempted, so a model gains by declining questions it would likely get wrong.

Does SimpleQA measure hallucination in long answers?

Not directly. The authors write that whether short-answer factuality correlates with long-form factuality remains an open research question.

What is SimpleQA Verified?

A 1,000-prompt cleaned version of SimpleQA by Haas, Yona, D'Antonio, Goldshtein and Das, which targets noisy labels, topical bias and redundant questions.

Share