Semantic similarity metrics for LLM outputs
Semantic similarity scores how close two texts are in meaning, usually by cosine similarity of embeddings. How it works, how to read the score and where it fails.
Semantic similarity is a score for how close two texts are in meaning, even when they share few words. For LLM outputs it is usually the cosine similarity between the embeddings of the output and a reference answer, and a higher score means closer meaning.
It is a reference-based check, so it needs a known good answer to compare against. It sits next to overlap scores such as BLEU and ROUGE, which count shared words, and LLM judges, which grade against a rubric. For the full picture, see LLM evaluation metrics.
How the score is computed
The common recipe has two steps. Embed both texts with the same model, then compare the two vectors. Reimers and Gurevych’s Sentence-BERT paper made this practical: it modifies BERT with siamese and triplet networks so sentence embeddings can be compared with cosine similarity.
The reason is cost. The paper says a pair-based BERT model, which reads both sentences together, needs about 50 million inference computations to find the most similar pair among 10,000 sentences, roughly 65 hours. Comparing precomputed embeddings brings that to about 5 seconds, with BERT’s accuracy kept, according to the authors.
The metric itself is a choice. The Sentence Transformers docs use cosine similarity by default and also offer dot product, negative Euclidean distance and negative Manhattan distance. If a model ends with a normalization step, the docs suggest the dot product, which gives the same results as cosine and runs faster. OpenAI recommends cosine similarity, notes that its embeddings are normalized to length 1, and says cosine and Euclidean distance give identical rankings.
One normalization detail is easy to miss. Google’s embeddings docs say that for gemini-embedding-001 you must normalize non-default (non-3072) dimensions yourself, while gemini-embedding-2 normalizes truncated dimensions automatically.
Three ways to score similarity
| Method | How it works | Trade-off |
|---|---|---|
| Bi-encoder and cosine | Each text is embedded on its own, then the vectors are compared. | Fast, and embeddings can be stored and reused. The Sentence Transformers docs say a bi-encoder produces embeddings independently for each text. |
| Cross-encoder | One model reads both texts together and outputs a score. | The Sentence Transformers retrieve and re-rank guide says cross-encoders have higher performance because they “perform attention across the query and the document”, but scoring thousands or millions of pairs would be slow. They recommend a bi-encoder to narrow to about 100 candidates, then a cross-encoder to re-rank. |
| Token-level matching | Matches each token embedding of one text with the closest token embedding of the other. | BERTScore works this way, using greedy matching, and returns precision, recall and F1 instead of one cosine value. |
For scoring one answer against one reference, speed rarely matters, so the cost argument favors bi-encoders mainly when you compare many pairs, as in a semantic cache or retrieval.
Reading the number
A similarity score is not a percentage or a probability. Its scale comes from the metric and the model.
- The range changes. In the Sentence Transformers example, cosine scores run from about -0.05 to 0.89, and the Manhattan example gives negative values (about -20.2 to 0), where closer to zero means more similar. Do not compare scores across metrics.
- Thresholds are local. A cutoff that works for one embedding model says nothing about another. The semantic caching page covers how vendors treat thresholds as something to tune against false positives. The same applies to a pass or fail rule in an eval: label a set of your own pairs as same or different, then pick the cutoff from where those scores fall.
- Task type can matter. Google’s docs list a
SEMANTIC_SIMILARITYtask type for gemini-embedding-001, described as embeddings optimized to assess text similarity. For gemini-embedding-2, the task instruction goes in the prompt text instead.
How humans define the target
The reference point for the field is the STS Benchmark, built from English semantic textual similarity shared tasks from 2012 to 2017 and described in the SemEval-2017 Task 1 overview. Annotators rate sentence pairs from 0, completely dissimilar, to 5, completely equivalent. Most pairs were rated by crowd workers, with five ratings per pair and the average as the gold score.
Systems are then scored by how well they agree with people: the task paper says performance is measured by the Pearson correlation of machine scores with human judgments. The same test works for your own metric. Collect human ratings on a sample of your outputs (see human evaluation) and check that the similarity score tracks them before you trust it.
Limits when scoring LLM outputs
- Similar is not correct. The BERTScore authors report that embedding matching finds it difficult to detect factual errors, such as a translation that says Tuesday where the reference says Monday. A fluent answer with a wrong figure can score high. Pair the score with checks on the facts, such as hallucination detection.
- It needs a reference. Open-ended answers with many valid forms do not have one best reference, so a judge or human review fits better.
- No model wins everywhere. The MTEB benchmark covers 8 embedding tasks, 58 datasets and 112 languages, and its authors report that no particular text embedding method dominates across all tasks. Test the embedding model on your domain and language.
- Scores are only comparable within a setup. Changing the embedding model, its dimensions or the metric changes the numbers, so record all three with each result and do not mix runs.
Using it in an eval pipeline
Compute the score in your own code with an embedding library, and keep the model name and metric with it. HoneyHive’s custom metrics guide shows how to log scores computed in application code onto a trace with enrich_span() or enrich_session(), so you can filter and monitor them. For what else to record around those calls, see LLM observability.
FAQ
What is semantic similarity?
A score for how close two pieces of text are in meaning, whatever words they use. In LLM work it is usually the cosine similarity between the embeddings of the two texts, as the Sentence-BERT paper set up.
How do you calculate semantic similarity between two sentences?
Embed each sentence with the same embedding model, then compare the vectors. The Sentence Transformers docs use cosine similarity by default and list dot product and Euclidean or Manhattan distance as alternatives.
Which is better, cosine similarity or dot product?
For normalized embeddings they rank pairs the same. OpenAI says its embeddings are normalized to length 1, so cosine can be computed with a dot product, and that the choice of distance function typically does not matter much.
What is a good semantic similarity score?
There is no universal one. The scale depends on the model and the metric: the Sentence Transformers docs show cosine scores in their example between about -0.05 and 0.89, and negative values for Manhattan distance. Pick a threshold by checking scores against pairs you have labeled yourself.
Does a high similarity score mean the answer is correct?
No. It measures closeness in meaning, not truth. The BERTScore authors say embedding matching finds it difficult to detect factual errors, such as Tuesday in place of Monday.