Skip to the lesson

ProductLesson 1 of 3

Part 1 of 3, 0 done

13 min

What a product manager does

An app's team has room to build a few things this season, and a list of fifty ideas. Someone has to choose. That's the product manager's job, and most ideas, even good-sounding ones, don't work out.

Ticket 01 · The guess

The idea that waited

In 2012 an employee at Bing, Microsoft's search engine, suggested a small change to the way ad headlines were shown. It was rated low priority, so it waited. When someone finally tested it, by how much did it raise Bing's revenue?

25%
0%50%
Words to know

Spot something wrong?

Tell us what looks off. No name or email needed.

What kind of problem?

0 / 1000

We'll also send this page, the lesson, ticket 1, so we can find it. Nothing else. Privacy

·
Sources and every number (3)

Sources

  • The Surprising Power of Online Experiments (Harvard Business Review, September–October 2017; the 2012 Bing ad-headline test, figures by the authors) · Ron Kohavi and Stefan Thomke, Harvard Business Review, 2012
  • Large Scale Experimentation at Bing (Bing blog, 8 August 2013; preface by Harry Shum, post by Ronny Kohavi) · Microsoft Bing, 2013
  • RICE: Simple prioritization for product managers (Intercom blog, 5 January 2018) · Sean McBride, Intercom, 2018

Every number

  • 12%hbr.org ·2012 · How much the 2012 ad-headline change raised Bing's revenue when tested: the article's authors' figure (they ran Bing's experiments).
  • $100M/yearhbr.org ·2012 · The authors' figure: the change was worth "more than" this much a year, in Bing's US revenue alone.
  • 6 monthshbr.org ·2012 · The idea "languished for more than" this long before anyone tested it, by the authors.
  • 250 experimentsbing.com ·2013 · Bing said "over" this many experiments were running on a typical day, in 2013.
  • 3intercom.com ·2018 · Intercom's RICE scale: massive impact.
  • 2intercom.com ·2018 · Intercom's RICE scale: high impact.
  • 1intercom.com ·2018 · Intercom's RICE scale: medium impact.
  • 0.5intercom.com ·2018 · Intercom's RICE scale: low impact.
  • 0.25intercom.com ·2018 · Intercom's RICE scale: minimal impact.
  • 100%intercom.com ·2018 · Intercom's RICE scale: high confidence.
  • 80%intercom.com ·2018 · Intercom's RICE scale: medium confidence.
  • 50%intercom.com ·2018 · Intercom's RICE scale: low confidence.
  • 4,000 studentsexample (example number: Our example: students a month the test-date reminders would reach.) · Our example: students a month the test-date reminders would reach.
  • 2example (example number: Our example: the reminders' impact on each student, on the RICE scale.) · Our example: the reminders' impact on each student, on the RICE scale.
  • 80%example (example number: Our example: how sure the team is about the reminders' reach and impact.) · Our example: how sure the team is about the reminders' reach and impact.
  • 2 person-monthsexample (example number: Our example: the work the reminders would take.) · Our example: the work the reminders would take.
  • 3,200example (example number: Our example: worked out: 4,000 students × 2 × 80% ÷ 2 person-months.) · Our example: worked out: 4,000 students × 2 × 80% ÷ 2 person-months.
  • 6,000 studentsexample (example number: Our example: students a month dark mode would reach.) · Our example: students a month dark mode would reach.
  • 0.5example (example number: Our example: dark mode's impact on each student, on the RICE scale.) · Our example: dark mode's impact on each student, on the RICE scale.
  • 100%example (example number: Our example: how sure the team is about dark mode.) · Our example: how sure the team is about dark mode.
  • 1 person-monthexample (example number: Our example: the work dark mode would take.) · Our example: the work dark mode would take.
  • 3,000example (example number: Our example: worked out: 6,000 students × 0.5 × 100% ÷ 1 person-month.) · Our example: worked out: 6,000 students × 0.5 × 100% ÷ 1 person-month.
  • 2,500 studentsexample (example number: Our example: students a month calendar sync would reach.) · Our example: students a month calendar sync would reach.
  • 2example (example number: Our example: calendar sync's impact on each student, on the RICE scale.) · Our example: calendar sync's impact on each student, on the RICE scale.
  • 50%example (example number: Our example: how sure the team is about calendar sync.) · Our example: how sure the team is about calendar sync.
  • 3 person-monthsexample (example number: Our example: the work calendar sync would take.) · Our example: the work calendar sync would take.
  • 2,500example (example number: Our example: worked out: 2,500 students × 2 × 50%, before dividing by the effort.) · Our example: worked out: 2,500 students × 2 × 50%, before dividing by the effort.
  • 833example (example number: Our example: worked out and rounded: 2,500 students × 2 × 50% ÷ 3 person-months.) · Our example: worked out and rounded: 2,500 students × 2 × 50% ÷ 3 person-months.
  • 1,500 studentsexample (example number: Our example: students a month a study-group chat would reach.) · Our example: students a month a study-group chat would reach.
  • 1example (example number: Our example: the chat's impact on each student, on the RICE scale.) · Our example: the chat's impact on each student, on the RICE scale.
  • 50%example (example number: Our example: how sure the team is about the chat.) · Our example: how sure the team is about the chat.
  • 4 person-monthsexample (example number: Our example: the work the chat would take.) · Our example: the work the chat would take.
  • 188example (example number: Our example: worked out and rounded: 1,500 students × 1 × 50% ÷ 4 person-months.) · Our example: worked out and rounded: 1,500 students × 1 × 50% ÷ 4 person-months.
  • 300 paid accountsexample (example number: Our example: what a school district offers to buy.) · Our example: what a school district offers to buy.
  1. Lessons›
  2. Inside a business›
  3. Product
  • ·Next: Talking to users and reading the data