Inter-annotator agreement and Cohen's kappa

Inter-annotator agreement measures how consistently two or more people (or a person and a model) assign the same label to the same items. Cohen’s kappa is the most common score for two raters: it takes the observed agreement and subtracts the agreement you would expect from chance, so 1.0 means complete agreement and 0.0 means no better than chance (scikit-learn docs). For LLM evaluation, it answers two questions: do your human labels agree with each other, and does your LLM judge agree with them?

Why percent agreement is not enough

Jacob Cohen introduced kappa in 1960 (Educational and Psychological Measurement 20(1)) because percent agreement cannot account for chance: raters may guess on some items (McHugh, 2012). When most outputs pass, two labelers who say “pass” almost every time will agree often without checking anything.

The same applies to LLM judges. Thakur et al. tested thirteen judge models and found that high percent agreement with humans can hide very different scores, so they recommend alignment metrics beyond simple percent alignment. Their best judges still fell well short of agreement between humans.

The formula

kappa = (p_o - p_e) / (1 - p_e)

p_o is the observed agreement and p_e is the agreement expected by chance, estimated from how often each rater uses each label (scikit-learn docs). The scikit-learn page gives a worked case: two annotators label five items as negative, positive or neutral, agree on four, and get a kappa of 0.6875. The raw agreement there is 0.8, which shows how much the chance correction lowers the score.

McHugh’s paper has a second case in which percent agreement was 0.94 and kappa 0.85, and notes that the greater the expected chance agreement, the lower kappa becomes (McHugh, 2012). The same paper advises calculating both percent agreement and kappa.

Reading the score

Kappa	Cohen’s label	McHugh’s label
0 or below	no agreement	none
0.01 to 0.20	none to slight	none
0.21 to 0.40	fair	minimal
0.41 to 0.60	moderate	weak
0.61 to 0.80	substantial	moderate (0.60 to 0.79)
0.81 to 0.90	almost perfect	strong
above 0.90	almost perfect	almost perfect

Labels from McHugh (2012); McHugh’s own bands sit slightly lower (0.21 to 0.39 minimal, 0.40 to 0.59 weak, 0.60 to 0.79 moderate, 0.80 to 0.90 strong). The first scale is Cohen’s, and the same bands are known as the Landis and Koch (1977) table. McHugh argues Cohen’s scale is too lenient for health research because it implies a score as low as 0.41 might be acceptable, and says any kappa below 0.60 indicates inadequate agreement.

Treat bands as a convention. The Landis and Koch table was meant for two annotators and two classes, offered no supporting evidence, and kappa’s size also depends on the number of categories and subjects (Fleiss’s kappa). Report the number, the raw agreement and the label counts, not just a band.

Variants

Statistic	Use it when
Cohen’s kappa	Two raters labeled every item (Fleiss’s kappa)
Weighted kappa (linear or quadratic weights)	Labels are ordered, such as 1 to 5 scores (scikit-learn)
Fleiss’s kappa	Any fixed number of raters; items may be rated by different people (Fleiss’s kappa)
Krippendorff’s alpha	Multiple raters and multiple possible ratings (McHugh)

Validating an LLM judge against human labels

An LLM judge grades another model’s output against a rubric, and Zheng et al. (2023) reported that strong judges such as GPT-4 reach over 80% agreement with human preferences, “the same level of agreement between humans”. The same paper names position, verbosity and self-enhancement bias. A judge is a classifier, so test it like one (see LLM evaluation).

Hamel Husain’s evals FAQ gives a workable procedure:

1. Label examples of the failure you want to detect. The FAQ suggests 100 to 200 examples for each failure mode, with at least 50 passing and 50 failing.
2. Check that the humans agree. If several people label, have them label the same examples independently, measure agreement with Cohen’s kappa, then discuss disagreements and update the rubric. The FAQ otherwise favors a single domain expert as a “benevolent dictator”.
3. Split the labels into train, dev and test sets. Tune the judge prompt on dev and hold the test set until you finish changing it.
4. Score the judge against the human labels with its true positive rate (how many real failures it catches) and true negative rate (how many good outputs it passes), and add kappa for a chance-corrected view.

Human agreement is the useful reference point. If your labelers agree with each other only weakly, a judge cannot be held to a higher standard against them, and the rubric is what to fix first. Thakur et al. make the same comparison, noting even strong judges fall short of inter-human agreement.

Computing kappa

from sklearn.metrics import cohen_kappa_score

cohen_kappa_score(human_labels, judge_labels)

This is the function documented by scikit-learn; pass weights="quadratic" for ordered scores.

Collecting human labels in HoneyHive

HoneyHive annotation queues organize events for human review. You add events from Traces with “Add to Queue”, or turn on queue automation so new events matching your filters are added as they arrive, then reviewers fill in the annotation fields in Review Mode. The fields come from human evaluators, which can return binary, numeric, categorical or free-text values.

The docs do not describe a built-in agreement metric or multiple reviewers per item. To compute kappa, bring the human and judge labels together yourself with the code above. Where the events come from is covered in LLM observability.

FAQ

What is a good Cohen's kappa?

There is no universal cutoff. Cohen's bands call 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect, while McHugh argues that "any kappa below 0.60 indicates inadequate agreement". The Landis and Koch table is "by no means universally accepted", so pick a target for your task and report the raw agreement next to it.

Can I use Cohen's kappa for more than two raters?

Not directly. Cohen's kappa assumes the same two raters labeled every item. Fleiss's kappa handles any number of raters, and Krippendorff's alpha covers multiple raters and multiple possible ratings.

Is percent agreement enough to validate an LLM judge?

No. Thakur et al. found that judges with high percent agreement can still give very different scores from humans, and recommend alignment metrics beyond simple percent alignment. Report a chance-corrected score and the judge's true positive and true negative rates.