🔍 Read the full analysis: FrontierCode: The Path From AI-Generated Code To A Pull Request on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Artificial Analysis published results for 23 models on FrontierCode, a coding benchmark designed to assess whether repository changes would pass maintainer-style review, not just tests. Claude Opus 5.5 led with a 52.2% score and cleared all blocking checks in 57.7% of attempts; the results do not establish that any model can reliably submit merge-ready changes without human review.
Artificial Analysis published FrontierCode results on October 12, ranking 23 AI models on whether their code changes meet repository-specific review criteria. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in only 57.7% of attempts—a result that underscores the gap between generating code and producing changes ready for a pull request.
FrontierCode evaluates work on tasks drawn from established software repositories. Unlike benchmarks that primarily count passing tests, it assesses patches across six dimensions: hidden tests, command checks, tests that must fail against the original code, adaptive tests for complex behavior, limits on file scope and diff size, and an LLM-based code review against maintainers’ criteria. Artificial Analysis ran the models using the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) served as the review judge.
A failed blocking check gives an attempt a score of zero, regardless of other successes; attempts that pass all blocking checks receive a score between 70% and 100%, with quality checks affecting the result. The benchmark also penalizes oversized changes, edits to unrelated files and unhelpful tests. Artificial Analysis reported that nine of the 100 Main-set tasks were not solved by any model on any attempt. The full benchmark contains 150 tasks, while the evaluation used the 100-task Main set.
After Opus, GPT-6.1 Sol scored 49.1% and Claude Fable 5.1 scored 49.0%. The next results included GPT-6 Astra at 47.0%, Claude Sonnet 5.5 at 45.8%, Xiaomi’s MiMo-V2.6-Pro at 45.4%, DeepSeek V4.1 Flash at 44.0%, Kimi K3 at 43.6% and Qwen3.8 Max at 43.2%. Artificial Analysis reported cost per task of $10.15 for Opus and $0.81 for Sol; those figures reflect this evaluation setup, not a universal price for using the models.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
From Passing Tests to Reviewable Patches
The benchmark focuses on a practical question for software teams: whether an AI-generated change is suitable for review and potential acceptance, rather than whether it merely passes a visible test suite. That distinction matters because maintainers must also judge whether a patch is appropriately scoped, whether its tests meaningfully catch regressions and whether it follows project-specific expectations.
The leading model’s 57.7% blocking-check pass rate is not a measure of how often it would be merged in a working organization. It is a result on this benchmark, under its task, agent and review setup. Still, it shows why the result cannot be treated as evidence that automated pull requests can routinely go straight to merge. Human review remains relevant, while the benchmark offers a way to compare models on dimensions closer to repository maintenance.
Cost also varies sharply among the reported systems. Artificial Analysis identified GPT-6 Luna, MiMo-V2.6-Pro, GPT-6.1 Sol and Opus as points on its cost frontier: no other model in the comparison was both cheaper and higher-scoring than each of those options. Moving from Sol to Opus meant a 3.1-point score difference alongside a reported cost increase from $0.81 to $10.15 per task. The trade-off may matter to teams weighing quality against evaluation-specific model costs, but it does not by itself establish which model is best for a particular codebase.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Repository Work
FrontierCode was built by Cognition with open-source maintainers and announced in June 2026. Its version 1.1 followed in July, adding rules against looking up solutions online. According to the source report, more than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task. The tasks are private to reduce the chance that a model has encountered the answers in training.
Artificial Analysis added the benchmark to its evaluations on October 12 and ran 23 models. The source report says nine models from six labs scored between 43% and 52%. It also compares results with the Artificial Analysis Intelligence Index: some models’ positions changed when judged on repository changes rather than general capabilities. For example, Claude Sonnet 5.5 was fifth on FrontierCode despite ranking second on the cited Intelligence Index comparison, while Grok 4.7 ranked 18th on FrontierCode. These comparisons describe the included models and measures; they do not show that general benchmarks are useless for other purposes.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
software repository review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The scores are benchmark results, not evidence that the models will perform at the same rate across all repositories, languages, team workflows or production pull requests. The supplied report does not give enough detail to independently assess how representative the 100 Main-set tasks are of day-to-day engineering work, or how closely the automated review judge agrees with human maintainers. The reported pass rates also depend on the benchmark’s agent harness, attempt limits and scoring rules.
Several models had no result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the current ranking does not compare every model discussed in the source report. It is also unclear from the material provided when those evaluations will be available or whether future benchmark versions will change task composition or scoring.
As an affiliate, we earn on qualifying purchases.
Results to Watch as Coverage Grows
The next useful development will be additional results for models not included in the October 13 comparison, especially those named as missing in the source report. Further evaluations could show whether the current ordering holds as coverage expands, though any new scores should be compared with attention to model versions and benchmark setup.
For now, teams can use FrontierCode as one source of evidence when comparing coding agents, alongside testing in their own repositories and human review. The published results do not remove the need to check generated changes for correctness, scope and maintainability before considering a pull request for acceptance.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated changes to software repositories using hidden and adaptive tests, command checks, scope limits and an LLM-based code review. A failed blocking check makes an attempt score zero.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s reported results, with a 52.2% score and a 57.7% rate of clearing all blocking checks.
Does the result mean AI pull requests can be merged without review?
No. The benchmark measures performance on its own tasks and setup. Its results do not establish reliable, unattended merging across real repositories, and the source report identifies human review as still relevant.
How many tasks and models were tested?
Artificial Analysis ran 23 models on the 100-task Main set. The full FrontierCode benchmark contains 150 tasks, according to the source report.
Which models were missing from the results?
As of October 13, 2026, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
