What is LiveCodeBench? Scenarios, versions and scoring
LiveCodeBench tests LLMs on new contest problems to limit contamination. See its four scenarios, release versions, how pass@1 works and what Pro adds.
LiveCodeBench is a coding benchmark for LLMs (for a shell-based one, see Terminal-Bench) that uses problems from programming contests and tags each one with its release date. The paper (Jain et al., first submitted March 2024) built it because HumanEval and MBPP were no longer enough to evaluate newer models. Because every problem has a date, you can score a model only on problems published after its training cutoff.
Where the problems come from
Problems come from weekly and biweekly LeetCode contests, AtCoder ABC contests and a small fraction of problems from Codeforces Div. 3 and 4 contests, all after April 2023 (paper, HTML). Difficulty follows each platform’s own ratings. The paper’s 511-problem set (May 2023 to May 2024) has 182 Easy, 206 Medium and 123 Hard problems, with 267 from AtCoder, 235 from LeetCode and 9 from Codeforces (paper, Table 1).
For tests, the paper uses platform tests where available. Otherwise GPT-4-Turbo writes input generators, with inputs capped at 100 per problem and about 17 tests per problem on average (paper).
The four scenarios
The paper defines four scenarios, which the repository exposes as codegeneration, selfrepair, codeexecution and testoutputprediction.
| Scenario | The model must | Source |
|---|---|---|
| Code generation | Write a solution from a problem statement | Paper |
| Self-repair | Fix an incorrect program using error feedback such as a failing test case or exception message | Paper |
| Code execution | Predict the output of a given function on a given input | Paper |
| Test output prediction | Give the expected output for a fixed set of test inputs from a problem | Paper |
How pass@1 is computed
For each problem the paper generates 10 candidates at temperature 0.2 and top_p 0.95. Pass@1 is estimated from those candidates as the share of problems solved, and code generation and self-repair need every test to pass (paper). The repository also computes pass@5 and is MIT licensed.
Contamination and date windows
The paper reports that DeepSeek models fall sharply on LeetCode problems released after about August 2023, and GPT-4-O drops on LeetCode problems after November 2023, its stated cutoff. The drop is mostly on LeetCode, with smoother results on AtCoder (paper).
The repository lets you filter by date: compute_scores.py takes --start_date and --end_date in YYYY-MM-DD format. A date window only helps if you know the model’s real training cutoff, which is a limit of the method.
Release versions
The repository README lists six releases, and --release_version defaults to release_latest.
| Release | Problems | Published through | Source |
|---|---|---|---|
| v1 | 400 | Mar 2024 | README |
| v2 | 511 | May 2024 | README |
| v3 | 612 | Jul 2024 | README |
| v4 | 713 | Sep 2024 | README |
| v5 | 880 | Jan 2025 | README |
| v6 | 1,055 | Apr 2025 | README |
All start in May 2023. The README listed no release after v6 when read. The project site overview still describes over three hundred problems from May 2023 to February 2024, so trust the repository for the current set. A score on v5 and a score on v6 cover different problems and are not directly comparable.
LiveCodeBench Pro
LiveCodeBench Pro (Zheng et al., submitted June 2025) is a separate, harder benchmark with problems from Codeforces, ICPC and IOI, updated continuously. Olympiad medalists label each problem by algorithmic category and analyze failed submissions. At the time of the paper, the best model reached 53% pass@1 on medium problems and 0% on hard ones without external tools.
Reading a score
- Name the version and date window. A v6 number and a v1 number cover different problems.
- Name the benchmark. LiveCodeBench and LiveCodeBench Pro are not interchangeable.
- Use the official source. The leaderboard shows Pass@1 with Easy, Medium and Hard columns. Third-party tables repeat the numbers at different dates, so check them against the official one. Scores are left out here because they change.
- Read it as one signal. It covers self-contained contest problems, not work in a real repository. For that, see SWE-bench.
Running it in your own evals
To track a coding model or agent over time, run the repository’s evaluation and log each run’s version, date window and settings with the results, so scores stay comparable. HoneyHive’s evaluation docs describe running and comparing experiments. For the wider method, see what LLM evaluation is.
FAQ
What does LiveCodeBench measure?
It measures how well a model solves programming contest problems, plus three related skills. The paper lists code generation, self-repair, code execution and test output prediction.
What is the difference between LiveCodeBench and LiveCodeBench Pro?
LiveCodeBench Pro is a separate benchmark with problems from Codeforces, ICPC and IOI, annotated by Olympiad medalists. The original draws on LeetCode, AtCoder and Codeforces (paper).
Which LiveCodeBench version should I compare?
Compare scores only from the same release version and date window. The repository lists releases v1 to v6 with different problem counts.
How is LiveCodeBench different from SWE-bench?
LiveCodeBench uses self-contained contest problems, while SWE-bench asks a system to patch real repositories and passes on repository tests.