What LLM Benchmarks Can and Cannot Tell You
LLM benchmarks abound, but the devil's in the details. We'll unpack their strengths, pitfalls, and how to read between the lines.
Every developer working with LLMs has a complicated relationship with LLM benchmarks. If you listen to the model providers, all of the benchmarks have been consistently climbing over the last year and a half. However, on the ground, the singularity seems no nearer1. How are we to reconcile these facts? I believe it all starts with having a realistic expectation of what LLM benchmarking can and cannot reasonably tell you.
Note that for the purposes of this discussion, we define a benchmark as a combination of a dataset containing individual datapoints and a method for quantifying the output of the model for each datapoint. This method can vary: it might involve measuring similarity to a 'golden' output or ranking outputs from different models in side-by-side comparisons.
The Good
Let’s lead with positivity and discuss what an LLM benchmark CAN tell you.
Absolute Performance
When it comes to narrow and unambiguously defined tasks, LLM benchmarks can genuinely deliver. For instance, consider a task like simple SQL query generation, where the goal is to transform a natural language query into a syntactically correct SQL statement that retrieves the appropriate data from a database. To disambiguate further, let’s say that this database has a known predefined schema and a predefined restriction on query types. In this case, the benchmark is well-defined because the correctness of the SQL query can be objectively measured. Either the query retrieves the correct data from the database, or it doesn’t. The restrictions on the schema and the query types further ensure that the benchmark is precisely defined. In the absence of this level of precision, it can be difficult to know whether or not any comparison between the benchmark task and the task you care about is apples-to-apples2. This level of precision might narrow the scope of the benchmark task but it can help ensure the fundamental validity of comparisons. On the flip side, a benchmark with general scope but uncertain validity is fundamentally useless.
Relative Performance
Now, let’s talk about relative performance—where things get a bit fuzzier but still useful. Even if you consider the absolute performance on a benchmark to be untrustworthy, benchmarks can still offer valuable insights when comparing models against each other. The reasoning? While a benchmark might have its flaws, it’s unlikely to systematically favor one model over another. This is a paradoxical benefit of the opaqueness of LLMs. It can be difficult to put your thumb on the scale for one model over another even if you tried. Therefore, even if you disregard the absolute performance metrics, the relative ranking of models on a particular benchmark can still provide meaningful insights.
Computational Performance
This is where benchmarks shine. Benchmarking is extremely well-suited for understanding the computational characteristics of different models, such as throughput and latency. The important caveat to note here is that it is important for benchmarks to characterize the variability in these metrics (i.e. the p90, p95, p99 values, etc.). It is generally unwise to take averages at face value. Tail events have disproportionate effect on software system operations and user satisfaction.
Custom Benchmarks: Your Secret Weapon
While generic benchmarks might offer a vague sense of broad capabilities, they often fall short when applied to specific domains. Creating a custom, domain-specific benchmark allows you to evaluate model performance against the exact tasks and data distributions that matter to your application. This tailored approach provides a more accurate and relevant measure of model effectiveness in your specific context than any generic benchmark could. By developing your own evaluation harness, you can ensure that the benchmark truly reflects the requirements of your system.
The Bad
While benchmarks have their upsides, there are some serious downsides that you need to be aware of. Here’s where things start to fall apart.
In-Sample vs. Out-of-Sample Dilemma
One of the big pitfalls in LLM benchmarking is in determining what exactly is meant to be “in-sample” for a type of task versus what is meant to be “out-of-sample”. For open-ended tasks, this distinction becomes a mess. Take a coding benchmark, for example: Is implementing k-means clustering using spectral embeddings in-sample because it’s a coding task? Or is it out-of-sample because it requires domain-specific math knowledge? The dataset description might say one thing, but the actual data can tell a different story. This is the classic spec vs. implementation debate all over again. With benchmarks, the larger the dataset, the harder it is to ensure every datapoint matches the spec. The result? You might be measuring something entirely different from what you intended.
Lack of Control in Computational Performance3
When you’re dealing with closed-source models, you’re basically at the mercy of the model provider. This means that anything not precisely specified within an SLA is liable to fall short of your expectations. If everything you care about is meticulously outlined within the SLA of your chosen model provider, then there’s little cause for concern. However, if this isn’t the case4, you should be aware of the potential pitfalls and prepare a contingency plan5.
The Skill Ceiling Problem in Pairwise Comparisons
A critical but often overlooked issue in benchmarking is the skill ceiling problem that arises in benchmarks that are graded via side-by-side comparisons between models. For a simple illustrative example, consider the prompt "What's up?"—a vague, open-ended question. No AI, no matter how advanced, will consistently score much higher than a 50% win rate on such prompts because of the inherent ambiguity and the lack of a definitive answer. What this means is that side-by-side benchmarks that contain datapoints that are either too easy or too ambiguous can underestimate the difference between strong models and weak models. This limitation reduces the usefulness of even gold-standard, human-evaluated Elo ratings, as meaningful distinctions between models become less apparent.
The Ugly
Unfortunately, the aforementioned problems might actually paint TOO rosy of a picture on LLM benchmarks because they are fundamentally solvable whereas there is one major potentially unsolvable problem lurking in the background.
Data Leakage: The Phantom Menace
Lack of transparency in dataset curation makes it unclear which benchmarks are being gamed by data leakage and which scores are legitimate. Data leakage occurs when data that is being used to evaluate a model also appears in the training set. This issue makes evaluations completely invalid with regard to assessing out-of-sample datapoints. This problem could be potentially alleviated with transparency over the datasets used to train these models. The thing to keep in mind, though, is that the only thing a statistical learning model can do is learn the data distribution of its training set. Therefore, the utility of such a model is strictly bounded by the usefulness of the training data. Therefore, the actual datasets used to train the models are the most value resource for model providers. For this reason, it’s unlikely we will see much transparency in this area. One should note that the lack of transparency in dataset curation isn’t just a minor issue—it’s a potential critical flaw in the fundamental idea of LLM benchmarking. Without transparency in how datasets are created and curated, it’s impossible to know whether a model’s performance is due to true generalization or just overfitting to leaked data.
Where Does This Leave Us?
Considering all of the above, one might be led to believe that LLM benchmarking is fairly hopeless outside of measuring throughput, latency and the like. I’m here to tell you that, well… That might be right! However, that shouldn’t stop anyone who really cares about LLM application performance from putting in a best-effort attempt. In the next part of this series, we’ll outline the key factors we use to determine the usefulness of various LLM benchmarks and why crafting your own custom benchmarks might be the most valuable tool in your toolkit.
1 A GPT-5 announcement would be an incredible jinx. I dare you, OpenAI.
2 For example, the success of a model on a SQL query task is going to be (at least in some way) a function of the distribution of the different types of SQL queries needed to solve the problems that are asked. If you don’t know the distribution of the types of queries needed to solve the problems in either the benchmark or the task you care about, then it can be impossible to determine the validity of using the benchmark results as a baseline.
3 As of the time of writing, OpenAI offers no SLA for ChatGPT or the API directly. An SLA is available for using OpenAI models through the Azure OpenAI Service, but it only guarantees uptime, not latency or throughput. The same situation holds true for Anthropic and AWS Bedrock (i.e., no direct SLA from Anthropic and an uptime-only SLA through AWS Bedrock).
4 For example, if the SLA only contains guarantees on p95 latency then that means it doesn’t contain guarantees on p99 latency. Do you care about p99 latency? If yes, then this might concern you. If no, then this is fine.
5 One open-source package that we’ve seen that solves this problem beautifully is LiteLLM. We’ve used this very successfully for all sorts of internal projects at HoneyHive.
