13 min
How a chatbot picks its next word
A chatbot doesn't look up your answer: it writes a likely next piece of text, then another, over and over. Watch it happen, and see why the same question can get two different answers.
Ticket 01 · The guess
How many pieces to choose from?
Words to know
Sources and every number (4)
Sources
- Language Models are Unsupervised Multitask Learners (Radford, Wu, Child, Luan, Amodei and Sutskever; section 2.3, the vocabulary) · OpenAI, 2019
- Understand and count tokens (Gemini developer documentation) · Google, 2026
- Prompt design strategies: temperature, topK and topP (Gemini developer documentation) · Google, 2026
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024), section 2.2 Confabulation · National Institute of Standards and Technology, 2024
Every number
- 50,257
openai.com ·2019 · The number of tokens on GPT-2's list (its vocabulary), from OpenAI's paper describing the model. - 100 tokens
google.com ·2026 · The token count in Google's rule of thumb for English text with its Gemini models. - 60–
80 words google.com ·2026 · About how many English words a hundred tokens make, by Google's rule of thumb for its Gemini models. - 70 words
(example number: Worked out: the middle of Google's range.) · Worked out: the middle of Google's range. - 70 words
(example number: Simplified example: a short reply, exactly one batch.) · Simplified example: a short reply, exactly one batch. - 100 tokens
(example number: Worked out from the example: one batch of words is about a hundred tokens.) · Worked out from the example: one batch of words is about a hundred tokens. - 150 words
(example number: Simplified example: one chatbot reply.) · Simplified example: one chatbot reply. - 214 tokens
(example number: Worked out from the example: the reply's words divided by seventy, times a hundred.) · Worked out from the example: the reply's words divided by seventy, times a hundred. - 188 tokens
(example number: Worked out from the example: the reply's words at eighty words for every hundred tokens.) · Worked out from the example: the reply's words at eighty words for every hundred tokens. - 250 tokens
(example number: Worked out from the example: the reply's words at sixty words for every hundred tokens.) · Worked out from the example: the reply's words at sixty words for every hundred tokens. - 1
(example number: Simplified example: the lab's starting randomness.) · Simplified example: the lab's starting randomness. - 0
(example number: Simplified example: no randomness, so the top-scoring token always wins.) · Simplified example: no randomness, so the top-scoring token always wins. - 2
(example number: Simplified example: the lab's highest randomness.) · Simplified example: the lab's highest randomness. - 5 tokens
(example number: Simplified example: all five of the lab's tokens may be picked.) · Simplified example: all five of the lab's tokens may be picked. - 3 tokens
(example number: Simplified example: only the three highest-scoring tokens may be picked.) · Simplified example: only the three highest-scoring tokens may be picked.