Skip to the lesson

Software startupLesson 8 of 8

Part 1 of 3, 0 done

14 min

Building with AI, and building AI into your product

In one early-2025 study, experienced developers using AI tools took longer, and still believed they'd been faster. Most developers now use AI tools, or plan to, and AI sits inside many products. Here's how to use it without shipping its mistakes, and what you owe customers once your product has AI in it.

Ticket 01 · The guess

Faster with AI?

July 2025. METR, a research group, studied 16 experienced developers doing 246arxiv.org ·2025 real tasks on big open-source projects they had worked on for years. For each task, chance decided whether AI tools were allowed. Before starting, the developers forecast that AI would cut their time by 24%arxiv.org ·2025.

When AI was allowed, how did the time to finish really change? Left of zero is faster, right of zero is slower.

0%
−50%50%

Spot something wrong?

Tell us what looks off. No name or email needed.

What kind of problem?

0 / 1000

We'll also send this page, the lesson, ticket 1, so we can find it. Nothing else. Privacy

·
Sources and every number (2)

Sources

  • Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker, Rush, Barnes and Rein; arXiv 2507.09089, v1 July 12, 2025, v2 July 25, 2025): a randomized trial of 16 experienced developers doing 246 tasks in mature open-source projects · METR, 2025
  • We are Changing our Developer Productivity Experiment Design (METR blog, February 24, 2026): a late-2025 follow-up to the early-2025 study; estimated speed-up for the original developers and new ones, with why METR calls the data weak and a lower bound · METR, 2026

Every number

  • 16arxiv.org ·2025 · Experienced developers in the trial, from large open-source projects they had worked on for years.
  • 246arxiv.org ·2025 · Real tasks from those projects; for each one, chance decided whether AI tools were allowed.
  • 24%arxiv.org ·2025 · Before starting, the developers forecast that allowing AI would cut the time to finish a task by this much.
  • 19%arxiv.org ·2025 · Allowing AI increased the time to finish a task by this much: AI tools slowed these developers down.
  • 20%arxiv.org ·2025 · After the study, the developers estimated that allowing AI had cut their time by this much.
  • 18%metr.org ·2026 · METR's late-2025 follow-up: estimated speed-up with AI for the developers who took part in the first study too (a wide range around it). METR calls the late-2025 data very weak evidence and likely an underestimate.
  1. Lessons›
  2. Industries›
  3. Software startup
  • ·Before this: Selling your software
  • ·Free guide: How software companies make money
Skip to the lesson

Software startupLesson 8 of 8

Part 1 of 3, 0 done

14 min

Building with AI, and building AI into your product

In one early-2025 study, experienced developers using AI tools took longer, and still believed they'd been faster. Most developers now use AI tools, or plan to, and AI sits inside many products. Here's how to use it without shipping its mistakes, and what you owe customers once your product has AI in it.

Ticket 01 · The guess

Faster with AI?

July 2025. METR, a research group, studied 16 experienced developers doing 246arxiv.org ·2025 real tasks on big open-source projects they had worked on for years. For each task, chance decided whether AI tools were allowed. Before starting, the developers forecast that AI would cut their time by 24%arxiv.org ·2025.

When AI was allowed, how did the time to finish really change? Left of zero is faster, right of zero is slower.

0%
−50%50%

Spot something wrong?

Tell us what looks off. No name or email needed.

What kind of problem?

0 / 1000

We'll also send this page, the lesson, ticket 1, so we can find it. Nothing else. Privacy

·
Sources and every number (2)

Sources

  • Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker, Rush, Barnes and Rein; arXiv 2507.09089, v1 July 12, 2025, v2 July 25, 2025): a randomized trial of 16 experienced developers doing 246 tasks in mature open-source projects · METR, 2025
  • We are Changing our Developer Productivity Experiment Design (METR blog, February 24, 2026): a late-2025 follow-up to the early-2025 study; estimated speed-up for the original developers and new ones, with why METR calls the data weak and a lower bound · METR, 2026

Every number

  • 16arxiv.org ·2025 · Experienced developers in the trial, from large open-source projects they had worked on for years.
  • 246arxiv.org ·2025 · Real tasks from those projects; for each one, chance decided whether AI tools were allowed.
  • 24%arxiv.org ·2025 · Before starting, the developers forecast that allowing AI would cut the time to finish a task by this much.
  • 19%arxiv.org ·2025 · Allowing AI increased the time to finish a task by this much: AI tools slowed these developers down.
  • 20%arxiv.org ·2025 · After the study, the developers estimated that allowing AI had cut their time by this much.
  • 18%metr.org ·2026 · METR's late-2025 follow-up: estimated speed-up with AI for the developers who took part in the first study too (a wide range around it). METR calls the late-2025 data very weak evidence and likely an underestimate.
  1. Lessons›
  2. Industries›
  3. Software startup
  • ·Before this: Selling your software
  • ·Free guide: How software companies make money