Tokenization: choosing the alphabet
How a tokenizer turns text into reusable pieces—and why its training is separate from the model’s.
In this lesson
A tokenizer defines the representation. The language model learns what tends to happen within that representation.
Two different training processes
Training a tokenizer and training a language model are different jobs. A tokenizer learns or defines a vocabulary and segmentation rules; the language model later learns relationships among the resulting IDs. Karpathy’s tokenizer course notes explicitly separate encoding and decoding from the neural-network stage.
An analogy is choosing the tiles for a word game before learning how to play. A set with common letter combinations needs fewer tiles per sentence, but a larger inventory. Once the tile set is fixed, changing its numbering without updating the model would send the wrong entries into the network.
The idea behind byte-pair encoding
A simplified BPE training process begins with small symbols, counts adjacent pairs, merges a frequent pair, and repeats. The learned merge order then helps encode new text. The 2016 subword paper applies this idea to neural translation; byte-level variants begin from bytes rather than whole words.
For an invented corpus containing “low low lower”, a training procedure might first merge l + o into “lo”, then “lo” + w into “low”. A later unseen word can still use smaller pieces. Real implementations also specify normalization, boundaries, special tokens, and tie-breaking; this miniature story omits those details deliberately.
The vocabulary changes the split.
Encoding is not guessing meaning
A fixed tokenizer generally follows its encoding algorithm; it does not ask the language model which split seems most meaningful. SentencePiece demonstrates another important design choice: learning subword units directly from raw sentences instead of requiring whitespace-separated words first. Kudo and Richardson, 2018.
Consider “New York” and “new york”. A case-sensitive tokenizer may produce different sequences even though a reader can associate both with the same place. Normalization can change this behavior, but can also remove distinctions. There is no universally ideal choice for every language, programming language, and task.
Practical consequences
More vocabulary entries can shorten sequences while enlarging input and output tables. Smaller pieces offer broad coverage but can require more steps to represent the same text. The tradeoff is architectural and statistical, not simply “bigger vocabulary is better”. Compare both sequence length and representation quality.
In the demo, try an empty string, a repeated word, a line break, and an emoji. Ask which behavior comes from the chosen segmentation rules. To explore a real BPE implementation afterward, Karpathy’s minbpe repository is a linked technical reference; this site does not copy its code or claim its tokenizer is running here.
Merges do not update during a chat
With a fixed toy rule a + b → ab, “abab” becomes ab · ab. Repeating “abab” in the prompt does not automatically add a new “abab” vocabulary entry. Learning a new tokenizer would be a separate preparation step.
Break text into tokens
A simplified longest-match tokenizer, not a product tokenizer.
The model invents a new token whenever it sees a new word.
A fixed tokenizer represents new strings using its existing pieces or fallback rules; ordinary inference does not grow the vocabulary.
Take it with you
- Tokenizer training precedes and differs from model training.
- BPE merges frequent adjacent pieces under defined rules.
- Tokenizer-specific counts cannot be inferred from this toy demo.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Training data: what the model gets to learn from
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Neural Networks: Zero to Hero ↗Andrej Karpathy · Living course; accessed 2026-11-13 · checked 13 Nov 2026
- Neural Machine Translation of Rare Words with Subword Units ↗Rico Sennrich, Barry Haddow, Alexandra Birch · 2016-08 · checked 13 Nov 2026
- SentencePiece ↗Taku Kudo, John Richardson · 2018-11 · checked 13 Nov 2026
- minbpe: byte-pair encoding reference implementation ↗Andrej Karpathy · Living repository; accessed 2026-11-13 · checked 13 Nov 2026