What is G-Eval?

G-Eval is a method for scoring generated text with a large language model instead of word-overlap formulas. A prompt states the task and criteria, the model writes its own evaluation steps, and a score comes from the probabilities of the rating tokens. It was introduced in a 2023 paper by Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu and Chenguang Zhu, later published at EMNLP 2023.

The authors started from a gap. Overlap metrics such as BLEU and ROUGE correlate poorly with human judgments on creative or varied output, and earlier LLM-based evaluators agreed with humans less than mid-sized trained neural metrics. G-Eval is a form of LLM-as-a-judge built to close that gap.

How G-Eval works

The paper describes three parts:

1. A prompt, “a natural language instruction that defines the evaluation task and the desired evaluation criteria.”
2. An automatic chain of thought. The model generates detailed evaluation steps from the task introduction and the criteria, and those steps go into the scoring prompt.
3. A scoring function. The model gets the prompt, the steps, the input and the text to judge, and fills in a form with a rating. Instead of keeping the single rating it returns, G-Eval weights each possible rating by its token probability: score = sum of p(s) x s.

The weighting matters because, the authors note, one digit usually dominates the score distribution and models tend to output integers even when asked for decimals. A probability-weighted score is a finer-grained value.

GPT-4 does not expose token probabilities, so the authors estimated them by sampling 20 times at temperature 1 with top_p 1. With GPT-3.5 they used temperature 0 and read the probabilities directly.

What the paper measured

The authors tested three benchmarks and compared scores with human ratings.

Average Spearman correlation with human judgments, as reported in the paper:

G-Eval with GPT-3.5 does much worse on QAGS than the GPT-4 version, which the authors read as consistency checking depending on model capacity. And the GPT-3.5 and GPT-4 versions score almost the same on Topical-Chat, so the authors treat that benchmark as relatively easy.

What each part contributed

In the paper’s SummEval ablation, G-Eval with GPT-4 dropped from a Spearman correlation of 0.514 to 0.500 without the chain of thought. Removing the probability weighting left Spearman at 0.502. Kendall-tau looked higher without it (0.446 against 0.418), but the authors note that plain integer scores create many ties, which inflate that statistic. On this benchmark each part adds about one to two hundredths of Spearman correlation.

Limits to know before you use it

• Bias toward model-written text. In the authors’ analysis, GPT-4 as evaluator always gave higher scores to GPT-3.5 summaries than to human-written ones, even where human judges preferred the human ones. Human agreement on that comparison was low (Krippendorff’s alpha of 0.07), and the authors say more research is needed. They also warn that using such an evaluator as a reward signal could cause self-reinforcement.
• Model capacity. The weaker backbone fell well behind on factual consistency, so re-validate when you change the judge model.
• Narrow evidence. The study covers summarization and dialogue in 2023 with the models of that time. Treat the correlations as a starting point and measure your own judge against human labels. Inter-annotator agreement shows how to check that those labels are reliable first.
• Cost. The sampling approach used with GPT-4 means 20 sampled completions per score.

G-Eval and other metrics

G-Eval sits with the model-based metrics in LLM evaluation metrics. BERTScore compares embeddings against a reference, while G-Eval needs no reference and judges against written criteria. For high-stakes decisions, keep human evaluation in the loop. The wider picture is in LLM evaluation.

Applying the pattern in HoneyHive

HoneyHive does not expose a G-Eval setting, but its LLM evaluator docs cover the same ingredients. You write the criteria and evaluation steps into the prompt, reference trace data with template variables such as {{ outputs.content }}, and pick a boolean, numeric or string return type. The docs’ example prompt lists its criteria and asks for an “Explanation:” before the rating, which is a plain version of reasoning before scoring. Sampling defaults to 10% of matching events.

FAQ

What does G-Eval stand for?

The name comes from the paper title, G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, by Liu, Iter, Xu, Wang, Xu and Zhu. GPT-4 was the backbone model the authors used.

Does G-Eval need a reference text?

No. The paper presents it as a reference-free evaluator, so it scores an output from the task description, criteria and input alone.

How accurate is G-Eval?

With GPT-4 the authors report a Spearman correlation of 0.514 with human ratings on summarization. That is an average over four quality dimensions on one dataset, so check any G-Eval-style judge against human labels on your own task.

Is G-Eval biased toward LLM-written text?

The authors found that GPT-4 as the evaluator gave higher scores to GPT-3.5 summaries than to human-written ones, even when human judges preferred the human summaries. They call it a preliminary study.