Skip to the lesson

Data and machine learningLesson 3 of 6

Part 1 of 3, 0 done

13 min

The test it hasn't seen

The bakery's new forecast was perfect on last month and way off on Monday. Here's why you never grade a model on its own homework.

Ticket 01 · The guess

The forecast that kept guessing high

Google Flu Trends estimated how many Americans were seeing a doctor for flu-like illness, from what people searched on Google. Researchers checked it against the CDC's own counts, week by week, starting in August 2011.

Out of 108 weeksscience.org ·2014, in how many weeks did it guess too high?

54 weeks
0 weeks108
Words to know

Spot something wrong?

Tell us what looks off. No name or email needed.

What kind of problem?

0 / 1000

We'll also send this page, the lesson, ticket 1, so we can find it. Nothing else. Privacy

·
Sources and every number (1)

Sources

  • The Parable of Google Flu: Traps in Big Data Analysis (Science 343(6176): 1203–1205, March 14, 2014) · Science (David Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani), 2014

Every number

  • 108 weeksscience.org ·2014 · Weeks checked against the CDC's counts, starting August 2011 (Lazer and others, Science, 2014).
  • 100 weeksscience.org ·2014 · Weeks, of those checked from August 2011, in which Google Flu Trends guessed too high (Lazer and others, Science, 2014).
  • 50 million search termsscience.org ·2014 · How many search terms the first version of Google Flu Trends picked its best matches from (Lazer and others, Science, 2014).
  • 1,152 data pointsscience.org ·2014 · How many data points of past flu levels the first version was fitted to (Lazer and others, Science, 2014).
  1. Lessons›
  2. The school›
  3. Data and machine learning
  • ·Before this: Predict tomorrow's iced coffees
  • ·Next: Fraud alert: two ways to be wrong
Skip to the lesson

Data and machine learningLesson 3 of 6

Part 1 of 3, 0 done

13 min

The test it hasn't seen

The bakery's new forecast was perfect on last month and way off on Monday. Here's why you never grade a model on its own homework.

Ticket 01 · The guess

The forecast that kept guessing high

Google Flu Trends estimated how many Americans were seeing a doctor for flu-like illness, from what people searched on Google. Researchers checked it against the CDC's own counts, week by week, starting in August 2011.

Out of 108 weeksscience.org ·2014, in how many weeks did it guess too high?

54 weeks
0 weeks108
Words to know

Spot something wrong?

Tell us what looks off. No name or email needed.

What kind of problem?

0 / 1000

We'll also send this page, the lesson, ticket 1, so we can find it. Nothing else. Privacy

·
Sources and every number (1)

Sources

  • The Parable of Google Flu: Traps in Big Data Analysis (Science 343(6176): 1203–1205, March 14, 2014) · Science (David Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani), 2014

Every number

  • 108 weeksscience.org ·2014 · Weeks checked against the CDC's counts, starting August 2011 (Lazer and others, Science, 2014).
  • 100 weeksscience.org ·2014 · Weeks, of those checked from August 2011, in which Google Flu Trends guessed too high (Lazer and others, Science, 2014).
  • 50 million search termsscience.org ·2014 · How many search terms the first version of Google Flu Trends picked its best matches from (Lazer and others, Science, 2014).
  • 1,152 data pointsscience.org ·2014 · How many data points of past flu levels the first version was fitted to (Lazer and others, Science, 2014).
  1. Lessons›
  2. The school›
  3. Data and machine learning
  • ·Before this: Predict tomorrow's iced coffees
  • ·Next: Fraud alert: two ways to be wrong