14 min
Building with AI, and building AI into your product
In one early-2025 study, experienced developers using AI tools took longer, and still believed they'd been faster. Most developers now use AI tools, or plan to, and AI sits inside many products. Here's how to use it without shipping its mistakes, and what you owe customers once your product has AI in it.
Ticket 01 · The guess
Faster with AI?
Sources and every number (2)
Sources
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker, Rush, Barnes and Rein; arXiv 2507.09089, v1 July 12, 2025, v2 July 25, 2025): a randomized trial of 16 experienced developers doing 246 tasks in mature open-source projects · METR, 2025
- We are Changing our Developer Productivity Experiment Design (METR blog, February 24, 2026): a late-2025 follow-up to the early-2025 study; estimated speed-up for the original developers and new ones, with why METR calls the data weak and a lower bound · METR, 2026
Every number
- 16
arxiv.org ·2025 · Experienced developers in the trial, from large open-source projects they had worked on for years. - 246
arxiv.org ·2025 · Real tasks from those projects; for each one, chance decided whether AI tools were allowed. - 24%
arxiv.org ·2025 · Before starting, the developers forecast that allowing AI would cut the time to finish a task by this much. - 19%
arxiv.org ·2025 · Allowing AI increased the time to finish a task by this much: AI tools slowed these developers down. - 20%
arxiv.org ·2025 · After the study, the developers estimated that allowing AI had cut their time by this much. - 18%
metr.org ·2026 · METR's late-2025 follow-up: estimated speed-up with AI for the developers who took part in the first study too (a wide range around it). METR calls the late-2025 data very weak evidence and likely an underestimate.