14 min
Training: learning from mistakes
Nobody types in a network's weights. Training nudges each one a little after every mistake, millions or billions of them, until the mistakes shrink.
Ticket 01 · The guess
How many knobs?
Words to know
Sources and every number (1)
Sources
- Language Models are Few-Shot Learners (Brown et al., arXiv:2005.14165; abstract and table 2.1) · OpenAI researchers, on arXiv, 2020
Every number
- 4 parameters
(example number: Worked out from the example: the bakery neuron's three weights and one threshold.) · Worked out from the example: the bakery neuron's three weights and one threshold. - 175 billion parameters
arxiv.org ·2020 · GPT-3's number of parameters, from the paper that introduced it. - 300 billion tokens
arxiv.org ·2020 · Tokens of text GPT-3 and its smaller versions were trained on, in total, per the same paper.