Real-SWE benchmark shows coding agents solve 38.8% of tasks

What happened

Specific Labs, a Y Combinator-backed applied AI research lab, published Real-SWE, a benchmark that runs coding agents against tasks lifted from private production codebases it licensed from real companies. The benchmark scored eight model-and-harness combinations across 640 rollouts on ten tasks, and the best performer, Anthropic's Fable 5.1 running inside Claude Code, resolved 38.8% of tasks. The result reached Hacker News on 12 September 2026, where it drew 157 points and 82 comments.

Real-SWE is a coding-agent benchmark built from ten tasks taken from private, production codebases at real companies, scored across eight frontier model-and-harness pairs. The top pair, Fable 5.1 on Claude Code, resolved 38.8% of tasks; six of the ten tasks stayed under a 15% resolution rate for every model tested.

What changed

Specific Labs licensed codebases from companies it describes as a Luma and Partiful competitor with more than 200,000 users and a top 100 App Store ranking, a consumer fintech platform processing more than 100,000 bank statements, and unnamed enterprise AI sales platforms. Each of the ten tasks was run eight times per model, for 640 scored rollouts, and resolution rate is reported as pass@1 averaged over those eight runs with 95% confidence intervals.

The full ranking:

1. Fable 5.1 on Claude Code: 38.8%
2. GPT-6 Astra on Codex CLI: 33.8%
3. Gemini 3.8 Flash on Gemini CLI: 31.2%
4. GLM 5.3 on Claude Code: 28.8%
5. Grok 4.6 on Grok Build: 23.8% (tied)
6. Muse Spark 1.3 on Muse Code: 23.8% (tied)
7. Kimi K3 on Kimi Code: 18.8%
8. GPT-5.6 Sol on Codex CLI: 16.2%

A typical Real-SWE task instruction runs 1,742 characters, against a median of 2,056 for Cognition's FrontierCode and 1,975 for DeepSWE. The reference solution to a Real-SWE task edits a median of 11 files, against 6 for FrontierCode and DeepSWE. Task environments connect agents to services including an AWS emulator, Docker, Kubernetes, GitHub, a Linear MCP server, PostgreSQL, MySQL, MongoDB, Redis, Slack, Intercom and Google Drive, and each task exposes only the services its workflow needs.

Short attempts did not do better than long ones: of 98 rollouts under ten minutes, 70 failed (71.4%), against 398 failures out of 542 rollouts of ten minutes or longer (73.4%). Specific Labs grouped failures with a taxonomy borrowed from DeepSWE: unverified assumption, missed requirement, integration error, regression and wrong file, and reports that different models fail in different ways within that breakdown. Estimated cost per rollout ranged from $2.50 for Gemini 3.8 Flash on Gemini CLI to $6.96 for Fable 5.1 on Claude Code, with Specific Labs flagging that the Grok 4.6 and Kimi K3 figures come from incomplete usage data and may run higher.

Why it matters for agent builders

The harness changes the result as much as the model does: GLM 5.3 scored higher running inside Claude Code than Grok 4.6, Muse Spark 1.3 or Kimi K3 scored in their own native harnesses. Anyone evaluating a coding agent for a codebase with real business rules should treat the model-and-harness pair as the unit to compare, not the model alone. The cost spread also matters at these resolution rates: Gemini 3.8 Flash resolved 31.2% of tasks at $2.50 a rollout, closer to GPT-6 Astra's 33.8% at $4.67 than the raw percentages suggest once cost is priced in.

The task-level spread is wide enough to change how a team reads any single resolution-rate headline. Real-SWE's own breakdown, requested from Specific Labs, shows some tasks (such as a multi-region sweep) resolved at 67.2% across all models and others (an analytics stream reducer task) resolved by no model in any of the eight runs allotted to it. A single aggregate score hides that a coding agent might be reliable on one class of enterprise change and unusable on another.

What to watch

Hacker News commenters questioned whether codebases described as private stay that way once an agent vendor runs a model against them, and whether some of today's frontier models have already seen similar internal patterns during training; Specific Labs has not published a response to that question. Separately, several commenters pointed to a known bug as a likely reason GPT-5.6 Sol finished last: OpenAI's Codex CLI repository tracks an open issue (Codex CLI 0.147.0) where GPT-5.6 Sol intermittently leaks internal tool-routing text as a literal <thinking> block in visible output, which derails a session until the user recovers it. Whether that bug affected Real-SWE's GPT-5.6 Sol rollouts specifically is not stated on the benchmark page.

Key takeaways

• Real-SWE scored eight model-and-harness pairs on ten tasks built from private production codebases, for 640 total rollouts.
• Fable 5.1 on Claude Code resolved the most tasks at 38.8%; GPT-5.6 Sol on Codex CLI resolved the fewest at 16.2%.
• Six of the ten tasks resolved below 15% for every model tested, and one task resolved for no model.
• Estimated cost per rollout ranged from $2.50 (Gemini 3.8 Flash) to $6.96 (Fable 5.1), so the top scorer was also the most expensive to run.
• The same model can score differently depending on its harness, which Real-SWE reports as a model-and-harness pair rather than a model score alone.

FAQ

What is Real-SWE?

Real-SWE is a benchmark from Specific Labs that scores coding agents on tasks copied or adapted from private, real-world production codebases the lab licensed from companies, rather than from public repositories or synthetic tasks.

Which model and harness scored highest on Real-SWE?

Fable 5.1 running inside Claude Code scored highest, resolving 38.8% of tasks, ahead of GPT-6 Astra on Codex CLI at 33.8% and Gemini 3.8 Flash on Gemini CLI at 31.2%.

Why did GPT-5.6 Sol score lowest on Real-SWE?

GPT-5.6 Sol on Codex CLI resolved 16.2% of tasks, the lowest of the eight pairs tested. Hacker News commenters linked the low score to an open Codex CLI issue in which GPT-5.6 Sol intermittently leaks internal tool-planning text into visible output and derails the session, though Specific Labs' benchmark page does not confirm this as the cause.

How much does it cost to run a Real-SWE rollout?

Specific Labs estimates rollout cost from $2.50 for Gemini 3.8 Flash on Gemini CLI up to $6.96 for Fable 5.1 on Claude Code, the same pair that scored highest on resolution rate.

Source: Real-SWE Discussion: Hacker News

Builders comparing coding agents and the MCP servers they connect to, such as the Linear MCP server used in Real-SWE's own task environments, can browse both in the MCP servers directory on aiagentslisting.com.