# Perplexity as a language model metric

> Perplexity is the exponentiated average negative log-likelihood a language model gives a text. Lower means less surprised.

Source: https://www.honeyhive.ai/learn/perplexity-metric (updated 2026-11-04)

Perplexity is the exponentiated average negative log-likelihood a language model assigns to a text. The [Hugging Face docs](https://huggingface.co/docs/transformers/perplexity) define it that way, and note it is equivalent to exponentiating the cross-entropy between the data and the model's predictions. Lower is better: the model was less surprised by the text.

Perplexity here means the language model metric, not the search company of the same name.

## The formula

For a tokenized sequence, the [Transformers docs](https://huggingface.co/docs/transformers/perplexity) give:

```text
PPL(X) = exp( -(1/t) * sum_i log p(x_i | x_<i) )
```

Here `p(x_i | x_<i)` is the probability the model gives token `i` after the tokens before it. Average the negative log of those probabilities, then exponentiate. The Evaluate card [says](https://raw.githubusercontent.com/huggingface/evaluate/main/metrics/perplexity/README.md) the range is 0 to infinity and a lower score is better.

Huyen (2019) [gives](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/) the intuition. With a base-2 logarithm, a model with 3 bits of entropy must choose among 2^3 = 8 equally likely options at each step, so its perplexity is 8. Read perplexity as the effective number of choices the model faces for each next token.

## Which models it applies to

Perplexity needs a probability for each token given the ones before it. The [Transformers docs](https://huggingface.co/docs/transformers/perplexity) say it applies to autoregressive (causal) models and is not well defined for masked models like BERT. The [Evaluate card](https://raw.githubusercontent.com/huggingface/evaluate/main/metrics/perplexity/README.md) also limits it to causal models. For masked-model comparisons of generated text, an embedding metric such as [BERTScore](/learn/bertscore) is a different tool.

## Computing it with a fixed context window

A model with a fixed [context window](/learn/context-window) cannot condition on a long text in one pass. The largest GPT-2 has a fixed length of 1024 tokens ([Transformers docs](https://huggingface.co/docs/transformers/perplexity)). The docs compare two ways around that:

- **Disjoint chunks.** Split the text into window-sized pieces and score each alone. This is fast, but the docs say it "will typically yield a higher (worse) PPL" because the model has less context at most steps.
- **Sliding window.** Move the window so each token has more context. It is closer to the true decomposition, but needs a forward pass for each token. A strided window is the practical compromise.

The docs show the size of the effect with GPT-2 large on WikiText-2. A stride of 1024 (no overlap) gives a perplexity of 19.44, about the 19.93 in the GPT-2 paper. A stride of 512 gives 16.44. Same model, same text, different number.

So a reported perplexity is only meaningful alongside the stride or context length. Huyen (2019) [says](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/) to specify the context length when reporting.

## Why two perplexity numbers rarely compare

| What changes | Effect | Source |
|---|---|---|
| Tokenizer | Perplexity is per token, so tokenization has a direct impact on it | [Transformers docs](https://huggingface.co/docs/transformers/perplexity) |
| Symbol level | A character-level score can be much lower than a word-level one without the model being better | [Huyen (2019)](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/) |
| Text and training data | The value depends heavily on what text the model was trained on; not comparable across models or datasets | [Evaluate card](https://raw.githubusercontent.com/huggingface/evaluate/main/metrics/perplexity/README.md) |
| Context or stride | A smaller stride usually reports a better number | [Transformers docs](https://huggingface.co/docs/transformers/perplexity) |
| Truncation | Inputs over the model's maximum length are truncated before scoring | [Evaluate card](https://raw.githubusercontent.com/huggingface/evaluate/main/metrics/perplexity/README.md) |

Huyen (2019) [adds](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/) that a comparison "does not signify a great deal unless we know how the text is pre-processed." Report the symbol level (word, character or subword) next to any score.

There is also a floor. Cross-entropy cannot fall below the entropy of the text itself, and that floor is usually unknown ([Huyen, 2019](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/)). A perplexity near the floor leaves little room to improve, but you rarely know where it is.

## What perplexity does not tell you

Perplexity scores how well a model predicts a particular text. It does not check that an answer is true, follows instructions or helps a user. [Huyen (2019)](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/) cites the XLNet authors' finding that improved language-modeling performance does not always improve downstream tasks. It is an intrinsic metric for the model, not a quality score for one response.

That is why teams pair it with task-level checks. [LLM evaluation](/learn/llm-evaluation) covers code checks, LLM judges and human review. Reference-based metrics such as [BLEU](/learn/bleu-score) and [BERTScore](/learn/bertscore) compare an output to a reference answer instead.

## Perplexity from an API

Some hosted APIs return token log probabilities for what they generate. The OpenAI Cookbook [explains](https://developers.openai.com/cookbook/examples/using_logprobs) that a logprob is the log of a token's probability (0.0 means 100%), and that perplexity can be found by exponentiating the negative of the average logprob, with higher perplexity meaning more uncertainty. The recipe is archived and may reference outdated models or APIs, so check your provider's current docs.

Here the number measures the model's own uncertainty about its output. A confident wrong answer still scores low, so treat it as one signal, not a verdict on correctness.

## Tracking it in an evaluation workflow

If you track perplexity across model or prompt changes, fix the text, tokenizer, context length and stride, and log them with each score. HoneyHive supports [client-side evaluators](https://docs.honeyhive.ai/v2/evaluators/client_side.md) that run scoring in your own code during experiments, which suits a metric that needs model log probabilities. [Python evaluators](https://docs.honeyhive.ai/v2/evaluators/python.md) run custom logic on HoneyHive's infrastructure, with a 4KB code limit, an approved package list and no file or network access. For watching production traces, see [LLM observability](/learn/llm-observability).

## FAQ

### What is perplexity in a language model?

The exponentiated average negative log-likelihood of a sequence, per the [Hugging Face docs](https://huggingface.co/docs/transformers/perplexity). It is also the exponential of the cross-entropy between the data and the model's predictions. Lower means the model assigned higher probability to the text.

### What is a good perplexity score?

There is no universal threshold. The Hugging Face Evaluate card [says](https://raw.githubusercontent.com/huggingface/evaluate/main/metrics/perplexity/README.md) scores depend heavily on the text the model was trained on and are not comparable between models or datasets. Compare runs only on the same text, tokenizer and context settings.

### Can I compare the perplexity of two different models?

Only if they share a tokenizer and you score the same text the same way. The [Transformers docs](https://huggingface.co/docs/transformers/perplexity) say tokenization has a direct impact on perplexity, and Huyen (2019) [notes](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/) that a lower character-level number does not mean a better model than a word-level one.

### Does low perplexity mean the output is correct?

No. It measures how well the model predicts a given text, not whether a generated answer is true or useful. Huyen (2019) [notes](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/) that better language modeling does not always improve downstream tasks.

### Can perplexity be computed for BERT?

Not in the standard way. The [Transformers docs](https://huggingface.co/docs/transformers/perplexity) say the metric applies to autoregressive language models and is not well defined for masked language models like BERT.

### Can I get perplexity from an LLM API?

If the API returns token log probabilities, yes. The OpenAI Cookbook [describes](https://developers.openai.com/cookbook/examples/using_logprobs) finding perplexity by exponentiating the negative average of the logprobs of the output tokens. That scores the model's own output, which tells you how uncertain it was.