Harvard study: AI coding agents hit a review bottleneck

Harvard study: AI coding agents hit a review bottleneck

Harvard researchers Fiona Chen and James Stratton studied 718 software firms and found that AI coding agents raise individual coding output sharply, but that gain does not show up in more finished features or fewer engineers. Their job market paper, "Artificial Intelligence in the Firm: Bottlenecks in Software Production" (current version 4 August 2026), traces the gap to code review. Kyle Orland reported the findings for Ars Technica on 9 October 2026, and the story reached 319 points and 275 comments on Hacker News.

AI coding agents, tools that write and submit code with minimal human direction, raised lines of code by 30%, commits by 20% and pull requests by 23% at firms that adopted them, the study found. None of that translated into a measurable rise in finished software: the rate at which firms closed Jira issues and epics did not change by a statistically significant amount after agent adoption.

What the study found

The paper uses a proprietary dataset from Jellyfish, an engineering analytics platform, covering 300 million work events across 725,938 workers at 718 firms from January 2021 to March 2026. The data combines five sources: HR records, GitHub coding activity, Jira issue tracking, Google Calendar meeting data, and recorded usage of GitHub Copilot, Cursor and Claude Code. Chen and Stratton compare firms before and after they adopted AI coding assistants (autocomplete-style tools) and separately AI coding agents, using a staggered difference-in-differences design that exploits the fact that firms adopted these tools at different times.

AI coding assistants produced smaller, mostly statistically insignificant gains: 12% more lines of code, 9% more commits, 5% more pull requests, with only the commit increase reaching significance. AI coding agents produced larger, significant gains across all three measures. For agents, the paper's confidence interval rules out a rise in firm-level software output larger than 12% of the baseline mean, well under the 30% individual coding gain, and the researchers found no shift toward larger or more complex Jira issues that might otherwise mask the output number.

The bottleneck shows up in the review stage. After agent adoption, the average time between a pull request being submitted and merged rose 49%. The share of pull requests sent back with changes requested nearly doubled, and comments per pull request rose 35%. Firms responded by shifting labor toward review: the share of workers performing code reviews rose 14%. By March 2026, 80% of the firms in the sample had adopted some form of AI code review tool, but those tools handled only 23.3% of review comments and 10.8% of pull requests, with human reviewers still doing most of the work. On employment, the paper reports a precise zero effect for AI agents on overall headcount (ruling out a decline greater than 2.9%) and a noisier zero on engineering headcount (ruling out a decline greater than 13.8%).

Why it matters for agent builders

A team that ships a coding agent expecting throughput to scale with lines of code written is measuring the wrong stage of the pipeline. This paper gives a sourced number for the stage that actually gates output: review. A 49% increase in review time and a near doubling of changes-requested pull requests means the review step, not the coding step, is where an agent rollout needs headcount, tooling or process changes to actually turn into more shipped work.

It also sets a baseline for AI code review tools. With 80% of studied firms already running one and agents still covering under a quarter of review comments, there is a large, mostly unclaimed share of review work for an agent or MCP-connected review tool to take on, if it can earn a human reviewer's trust on quality rather than just volume.

What to watch

The paper's data ends in March 2026, and the authors note that agent capabilities have changed materially since that cutoff, with adoption reaching 95% of the firms in the sample by then. Whether later-generation agents write code that needs less review, and whether firms that have had more time with agents learn to restructure the review step rather than just add reviewers to it, are the open questions the next version of this research, or a follow-up study, would need firm-level data through late 2026 to answer.

Key takeaways

• AI coding agents raised lines of code by 30%, commits by 20% and pull requests by 23% across 718 firms studied by Harvard researchers Fiona Chen and James Stratton.
• Firm-level software output, measured by Jira issue and epic resolution, did not rise by a statistically significant amount after agent adoption.
• Code review time rose 49%, pull requests sent back for changes nearly doubled, and review comments per pull request rose 35%.
• Firms shifted labor toward review, with the share of workers doing code review up 14%, while AI review tools still handled under a quarter of review comments by March 2026.
• The study finds no statistically significant employment effect from AI coding agent adoption, at either the overall or engineering-staff level.

FAQ

What is the Harvard AI coding agents study?

It is a job market paper by Fiona Chen and James Stratton of Harvard University, "Artificial Intelligence in the Firm: Bottlenecks in Software Production," current version 4 August 2026. It uses engineering analytics data from Jellyfish covering 718 firms and 300 million work events from January 2021 to March 2026 to measure how AI coding assistants and AI coding agents affect coding output, software output and employment.

Why doesn't more code written by AI agents mean more software shipped?

The study attributes the gap to a bottleneck in code review. After firms adopted AI coding agents, average pull request review time rose 49%, the share of pull requests sent back for changes nearly doubled, and review comments per pull request rose 35%, which absorbed the extra code the agents produced before it could turn into finished, deployed features.

Did the study find AI coding agents reduced software engineering jobs?

No. The researchers found a precise zero effect of AI coding agent adoption on overall firm employment, ruling out a decline greater than 2.9%, and a noisier zero effect on engineering-specific employment, ruling out a decline greater than 13.8%. They also found firms shifted a larger share of their existing engineers into code review rather than cutting headcount.

Source: Ars Technica

Discussion: Hacker News

Teams sizing up where an agent rollout will actually bottleneck can compare coding and review tools in the coding category on AI Agents Listing.