Follow a token
Travel through the numerical pipeline, from text pieces to a probability distribution. Use the worked pipeline alongside the lessons.
By the endDescribe what changes at each stage of a decoder-style transformer.
Start path: Tokenization: choosing the alphabet →7 lessons · 42 min total
Mark lessons as read to save your place on this browser.
- 16 MIN READING · Not read yet
Tokenization: choosing the alphabet
Turn characters into vocabulary entries.
Read the lesson → - 26 MIN READING · Not read yet
Embeddings: giving tokens coordinates
Look up a vector for each entry.
Read the lesson → - 36 MIN READING · Not read yet
Positions: giving order to the sequence
Represent where each entry appears.
Read the lesson → - 46 MIN READING · Not read yet
Attention: a computed weighted mixture
Mix information from available positions.
Read the lesson → - 56 MIN READING · Not read yet
Logits: scores before probabilities
Project a representation into candidate scores.
Read the lesson → - 66 MIN READING · Not read yet
Softmax: turning scores into a distribution
Normalize the scores into probabilities.
Read the lesson → - 76 MIN READING · Not read yet
Sampling: choosing the next token
Choose a token under a decoding rule.
Read the lesson →