Skip to the lesson

How AI worksLesson 3 of 6

Part 1 of 3, 0 done

14 min

Training: learning from mistakes

Nobody types in a network's weights. Training nudges each one a little after every mistake, millions or billions of them, until the mistakes shrink.

Ticket 01 · The guess

How many knobs?

Every weight and threshold inside a model is a parameter: a number training has to set. The bakery's one-neuron rule has 4 parametersexample (example number: Worked out from the example: the bakery neuron's three weights and one threshold.). GPT-3, a language model OpenAI described in 2020, has how many?

250 billion parameters
0500 billion
Words to know

Spot something wrong?

Tell us what looks off. No name or email needed.

What kind of problem?

0 / 1000

We'll also send this page, the lesson, ticket 1, so we can find it. Nothing else. Privacy

·
Sources and every number (1)

Sources

  • Language Models are Few-Shot Learners (Brown et al., arXiv:2005.14165; abstract and table 2.1) · OpenAI researchers, on arXiv, 2020

Every number

  • 4 parametersexample (example number: Worked out from the example: the bakery neuron's three weights and one threshold.) · Worked out from the example: the bakery neuron's three weights and one threshold.
  • 175 billion parametersarxiv.org ·2020 · GPT-3's number of parameters, from the paper that introduced it.
  • 300 billion tokensarxiv.org ·2020 · Tokens of text GPT-3 and its smaller versions were trained on, in total, per the same paper.
  1. Lessons›
  2. The school›
  3. How AI works
  • ·Before this: One neuron, one decision
  • ·Next: How a computer sees a photo
Skip to the lesson

How AI worksLesson 3 of 6

Part 1 of 3, 0 done

14 min

Training: learning from mistakes

Nobody types in a network's weights. Training nudges each one a little after every mistake, millions or billions of them, until the mistakes shrink.

Ticket 01 · The guess

How many knobs?

Every weight and threshold inside a model is a parameter: a number training has to set. The bakery's one-neuron rule has 4 parametersexample (example number: Worked out from the example: the bakery neuron's three weights and one threshold.). GPT-3, a language model OpenAI described in 2020, has how many?

250 billion parameters
0500 billion
Words to know

Spot something wrong?

Tell us what looks off. No name or email needed.

What kind of problem?

0 / 1000

We'll also send this page, the lesson, ticket 1, so we can find it. Nothing else. Privacy

·
Sources and every number (1)

Sources

  • Language Models are Few-Shot Learners (Brown et al., arXiv:2005.14165; abstract and table 2.1) · OpenAI researchers, on arXiv, 2020

Every number

  • 4 parametersexample (example number: Worked out from the example: the bakery neuron's three weights and one threshold.) · Worked out from the example: the bakery neuron's three weights and one threshold.
  • 175 billion parametersarxiv.org ·2020 · GPT-3's number of parameters, from the paper that introduced it.
  • 300 billion tokensarxiv.org ·2020 · Tokens of text GPT-3 and its smaller versions were trained on, in total, per the same paper.
  1. Lessons›
  2. The school›
  3. How AI works
  • ·Before this: One neuron, one decision
  • ·Next: How a computer sees a photo