FrontierCode 2.0

Evaluate agents
HoneyHive Engineering

FrontierCode 2.0 is Cognition's coding benchmark that grades whether an agent's change is mergeable. How tasks are built, scored and where it differs from 1.0.

FrontierCode 2.0 is a coding benchmark from Cognition that asks whether an agent’s output is mergeable, not only whether it passes tests. Cognition’s announcement, dated October 27, 2026, describes it as the next generation of its FrontierCode series and as its best attempt yet to evaluate user experience. Where SWE-bench checks a patch against tests, FrontierCode also grades whether the change fits the codebase and the unstated expectations of a maintainer.

What it measures

Cognition says the benchmark looks at functional correctness, how the agent interprets and expands the user’s prompt, and design and scoping choices that integrate with an existing codebase. A central question is whether a model can infer constraints the prompt never states.

How the tasks are built

More than 20 open-source developers each spent over 40 hours per task, following largely the same method as FrontierCode 1.0. Tasks come from 55% more repositories than 1.0 and include security patches and performance optimizations. In 80% of tasks the reference solution differs from the upstream pull request, up from 60% in 1.0. The post does not give a total task count.

How it is graded

A score is a weighted aggregate of rubric items. A solution that fails a blocking criterion gets 0. Blocking criteria cover thoroughness, core behavior, regressions, tests, lint and typecheck, and scope, among others such as backwards compatibility, compilation and test integrity. Cost is the mean spend per rollout at list API pricing.

Compared with 1.0, tasks have 43% more blocking criteria. Compared with 1.1, reference solutions modify 67% more files and 89% more lines, and codebases are 25% larger. Scope rules were loosened, so the scope blocker failure rate fell from 9.0% in 1.1 to 3.8%, while thoroughness blockers rose roughly threefold in count.

Scores

Cognition’s launch table (October 2026) shows each model at its best reasoning effort, on version 1.1 and on 2.0. The top four:

Model 1.1 2.0
Claude Opus 5.5 54.6% 47.3%
GPT-6 Astra 53.3% 44.0%
Grok 4.7 47.6% 28.9%
Gemini 4 Argon 47.5% 27.6%

Cognition calls Opus 5.5 the top model at launch and the benchmark far from saturated. Scores move quickly, so treat these as a snapshot.

FrontierCode-AA

Artificial Analysis runs its own version, FrontierCode-AA 2.0. Its methodology uses 132 private tasks in real open-source repositories, so models cannot train on them. Each task runs three times in the Stirrup harness, capped at 500 turns, and a verifier runs the task’s tests plus a rubric of blocking and non-blocking checks, with an LLM judge (GPT-6 Luna at medium effort). An attempt scores 0 if it fails a blocking check or a trajectory scanner flags it for looking up the solution. The reported score is the mean over three attempts.

On November 20, 2026 its leaderboard had Claude Fable 5.2 (High) at 49.1%, GPT-6.2 Astra (Xhigh) at 49.0% and Claude Fable 5.2 (Max) at 48.8%. These numbers are not comparable with Cognition’s table, since the task sets, harness and judge differ.

What it found about agents

Cognition reports that models often fail to infer intent beyond the explicit spec and introduce bugs that break constraints a careful engineer would have inferred. Thoroughness has the highest blocker failure rate at every effort level, 26.0% at low effort and 15.3% at max. Scores can also fall at higher reasoning effort. Cognition names scope blockers as the primary factor, since models write more code and touch more files as effort rises, with occasional cheating by pulling the solution commit and overthinking as smaller factors.

Reading the results

A model that tops FrontierCode may still miss your repository’s conventions. For your own agents, build a golden dataset from real tasks, grade with a rubric that separates blocking from non-blocking checks, and use LLM-as-a-judge with spot checks by a person. Other coding and agent tests are covered in agent benchmarks, Terminal-Bench and agentic coding.

Running a similar evaluation in HoneyHive

You can import a dataset of your own coding tasks and compare runs as described in the experiments guide.

FAQ

Who made FrontierCode 2.0?

Cognition published it on October 27, 2026 as the next generation of its FrontierCode benchmark series.

How is FrontierCode 2.0 scored?

Cognition says the score is a weighted aggregate of rubric items, and a solution that fails any blocking criterion gets 0.

What is FrontierCode-AA?

It is Artificial Analysis's implementation of FrontierCode 2.0: 132 private tasks in real open-source repositories, run three times each in its Stirrup harness.

Is FrontierCode 2.0 saturated?

Cognition calls it far from saturated, with the top model at launch scoring only 47%.

Share