Armature's coding-agent tool study finds 42% agreement

What happened

Armature, a firm that sells growth services to developer tools, published a study on 3 September 2026 that it calls the largest study to date of how coding agents choose third-party tools. The team ran 16,893 sessions across Claude Code, Codex and Cursor, using 1,163 prompt variations on 75 repositories written in 10 languages, with fake company names and real lockfiles. A simulated human played by Gemini 3.7 Flash sat in the loop, and a second Gemini instance judged which sessions were valid and which tool won each one.

Armature found that Claude Code, Codex and Cursor pick the same tool in only 42% of matched sessions. Each agent leans on a different source: Cursor checks the web in about two-thirds of sessions, Codex searches the web in 94% of cases, and Claude Code relies mostly on its own priors, searching the web in roughly 30% of sessions.

What changed

• Repository context flips the winner. For the same "email this invoice to the user" prompt, TypeScript repos picked Resend 55 of 89 times, Python repos picked Sendgrid 22 of 24, Go repos picked Postmark 20 of 24, and Java repos picked Azure ACS 22 of 23.
• Being mentioned is not being picked. PayPal was cited in 139 sessions and never won a single one, with Stripe taking 124 of those 139. LangChain was the most-cited framework at 194 mentions but was picked only 4 times. Supabase was the most-cited database at 242 mentions and still lost to Neon.
• Vendor page copy changes outcomes. Mailgun lost to Postmark after agents read "1-day retention" on its free plan, and Supabase lost head-to-head comparisons where the agent only wanted a database, not bundled BaaS features.
• Some categories are close to monopolies: Stripe won 9 in 10 payment sessions, Neon took 66% of database sessions, and Amazon S3 took 45% of file storage sessions.
• Claude Code builds a solution in-house almost twice as often as the other two agents, 19% of sessions versus 10%.
• Codex narrows its web search with site: operators (for example, site:auth0.com) in 9 of 10 search sessions. Claude Code searches the web far more when its priors are weaker, reaching about 80% of sessions in newer categories like sandboxes.

Why it matters for agent builders

If you build an agent, MCP server or skill that a coding agent should pick up, showing up in a training corpus or a search result does not translate into a win. The study's category breakdowns show that clear, specific pricing and feature pages beat brand recognition; that's how Postmark took the retention-sensitive case from Mailgun and how Stripe kept converting mentions of PayPal into its own wins. For builders listed on AI Agents Listing, the practical read is to say plainly what a product does and does not include, since agents appear to read that copy closely before implementing anything.

The 42% agreement rate also matters for anyone shipping across multiple coding agents: a tool that wins consistently in Claude Code sessions can lose the same task in Cursor or Codex, so testing against one agent alone will not tell you how the others will decide.

What to watch

Armature kept only 5,292 of the 16,893 sessions as valid for this first release, spanning 51 codebases and 18 sectors, and says it may publish more from the remaining runs later. The Hacker News discussion of the study, at 262 points and 123 comments, has run mostly on open-weight models and agent provider lock-in rather than the per-category leaderboards, so it's still open how vendors will respond to the incentive to clean up their pricing and feature pages.

Key takeaways

• Claude Code, Codex and Cursor pick the same third-party tool in only 42% of matched sessions across Armature's 16,893-session study.
• Being cited doesn't mean winning: PayPal was mentioned 139 times but never won, and Stripe took 124 of those 139 sessions.
• The programming language of a repository can change which vendor wins on an identical task, as shown with email providers across TypeScript, Python, Go and Java repos.
• Stripe, Neon and Amazon S3 each dominate their category, winning 9 in 10, 66% and 45% of sessions respectively.
• Armature has published the raw session traces, prompts and code diffs for anyone to review.

FAQ

What is the Armature coding-agent tool study?

It is a study Armature published on 3 September 2026 analyzing 16,893 sessions across Claude Code, Codex and Cursor to see which third-party tools each agent picks when implementing a feature, using 1,163 prompt variations across 75 repositories in 10 languages.

Do Claude Code, Codex and Cursor pick the same tools?

No. They agree on the same tool in only 42% of sessions. Cursor checks the web in about two-thirds of sessions, Codex searches the web in 94% of cases (usually narrowed with site: operators), and Claude Code searches the web in only about 30% of sessions, relying more on its own priors.

Where can I see the raw data from the study?

Armature published the full traces, leaderboards and category results at armature.tech, including the user prompts, agent thinking traces and the actual code diffs from each session.

Source: Armature Discussion: Hacker News

The Coding category on AI Agents Listing tracks tools like the ones this study measured.

• AI Agents Listing team