HarnessTax finds coding harness moves cost, not success

UC Berkeley's Sky Lab and Arena published HarnessTax on 16 September 2026, a study that runs seven language models through three coding agent harnesses, Claude Code 2.1.224, Codex CLI 0.146.0 and Pi 0.85.1, on the same 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0. The authors are Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia. The paper reached 216 points and 87 comments on Hacker News.

HarnessTax is a UC Berkeley Sky Lab and Arena study showing that a coding agent's harness, the program that manages its tools, prompts and turn loop, changes cost far more than it changes task success. Claude Fable 5 solved 97.8% of SWE-bench Lite attempts in Claude Code and 96.7% in Pi, but Claude Code cost $1.33 per attempt against $0.67 in Pi.

What changed

• The study ran 21 model and harness pairs (seven models across three harnesses), each pair three times per task, 30 tasks per benchmark, capped at 100 agent turns, using each harness's own high effort setting, with costs computed from a fixed direct-API price list dated 1 September 2026.
• The harness's effect on success rate stayed within about plus or minus 2% on SWE-bench Lite and about plus or minus 5% on Terminal-Bench 2.0. Across shared models, Claude Code cost about 2.0 times Pi and 1.6 times Codex CLI on SWE-bench Lite, using geometric means, and about 1.5 times Pi on Terminal-Bench 2.0.
• Pi, a minimal open-source harness with four tools (read, write, edit, bash), reached the Pareto frontier of cost against success on both benchmarks. Its first model call on SWE-bench Lite carried 2,873 characters of tool schemas and 1,972 input tokens. Codex CLI's first call carried 18,114 characters across an average of 7.4 tools and 11,308 tokens. Claude Code's first call carried 76,995 characters across 23 tools and 27,011 tokens.
• On Terminal-Bench 2.0, GPT-5.6 Sol reached an 83.3% success rate in Pi against 78.9% in Codex CLI, at $0.42 per attempt against $0.76. Across six Anthropic and OpenAI models on both benchmarks, a harness other than the model's own vendor harness produced the highest observed success rate in nine of twelve comparisons, including Sonnet 4.6 solving 68.9% of SWE-bench Lite attempts in Codex CLI against 66.7% in Claude Code at a similar cost.

Why it matters for agent builders

Someone wiring a coding agent into a pipeline is choosing a harness as much as a model, and HarnessTax puts a number on that choice. Claude Code ships 23 tools by default, and every one of those tool schemas rides along in the first call and every call after it, billed as a cache write when the prompt is cold and a cache read otherwise. A team running short, well-defined tasks such as a bug fix with an existing test suite pays for tool definitions it may never call, while a long session that needs subagents, hooks or MCP connections, none of which HarnessTax measured, may need exactly that scaffolding to finish the job. The same accounting applies to MCP servers: connecting several while using one adds its own schema weight to that same first call, on top of what the study measured for the harness alone.

What to watch

The study covers two public benchmarks the models may have already seen in training, short tasks capped at 100 turns and averaging about 15, and direct-API pricing that does not apply on a Claude Pro or Max subscription, where the constraint is a weekly usage window rather than a per-call bill. It is also self-published without peer review, with API access sponsored by Arena and Anthropic credits provided by the Laude Institute. The authors argue the fix is not picking Pi over Claude Code by default but testing a harness against a workload before adopting it, and they close by saying a harness should adapt to the task automatically rather than leave the choice to the user by hand.

Key takeaways

• HarnessTax ran seven models through three coding agent harnesses on SWE-bench Lite and Terminal-Bench 2.0 and found harness choice moves cost by up to 5x while moving success rate by only about 2 to 5 percentage points.
• Claude Code's default 23 tools carried 76,995 characters of tool schemas into the first model call on SWE-bench Lite, against 2,873 characters for Pi's four tools and 18,114 for Codex CLI's average of 7.4 tools.
• Pi, a minimal open-source harness, reached the Pareto frontier on both benchmarks despite offering only read, write, edit and bash.
• An alternative harness beat the model's own vendor harness in nine of twelve comparisons across six Anthropic and OpenAI models.
• The study did not test subagents, hooks, MCP connections or memory across sessions, features that add their own cost on top of what it measured.

FAQ

What is HarnessTax?

HarnessTax is a study published on 16 September 2026 by UC Berkeley's Sky Lab and Arena that measures how much a coding agent's harness, as opposed to its underlying model, affects cost and task success. It ran seven models through Claude Code, Codex CLI and Pi on SWE-bench Lite and Terminal-Bench 2.0.

Does a bigger harness make a coding agent better at its job?

Not on the tasks HarnessTax tested. Claude Fable 5 solved 97.8% of SWE-bench Lite attempts in Claude Code against 96.7% in the four-tool Pi harness, a gap inside the study's confidence interval, while Claude Code cost about twice as much per attempt.

What is Pi?

Pi is a minimal, open-source, MIT-licensed coding agent harness that ships with four tools: read, write, edit and bash. HarnessTax found it reached the Pareto frontier of cost against success on both benchmarks it tested.

Source: HarnessTax

Discussion: Hacker News

Developers comparing coding agents and their harnesses before wiring one into a pipeline can browse the options in the coding category on AI Agents Listing.