Explore the lessons.
Find a topic below or search for a question. New here? The beginner path gives you five lessons in a useful order.
Text & data
Before a model sees language, someone decides what counts as data.
Tokens: the pieces a model reads
A model works with a vocabulary of numbered pieces. Those pieces do not have to line up with words.
Tokenization: choosing the alphabet
How a tokenizer turns text into reusable pieces—and why its training is separate from the model’s.
Training data: what the model gets to learn from
A dataset is a designed mixture of examples, not a perfect copy of the world.
Data quality: more than removing typos
Duplicates, contamination, format errors, and missing perspectives affect what a model learns and how it is measured.
Inside the model
Turn a sequence of numbers into a context-sensitive representation.
Embeddings: giving tokens coordinates
A lookup turns discrete IDs into vectors that a neural network can transform.
Positions: giving order to the sequence
The same tokens in a different order can mean something different. A model needs a way to represent that order.
Parameters: the numbers learning changes
Weights shape a model’s computation. They are different from the text you put in a prompt.
Neural networks: composing simple calculations
Large models combine many ordinary numerical operations into a trainable function.
Transformers: a repeated pattern of computation
Token representations repeatedly exchange information and transform it before producing next-token scores.
Attention: a computed weighted mixture
Queries and keys determine weights; those weights combine values from permitted positions.
Causal masking: no peeking at the answer
A next-token learner must only use the prefix available at the moment it predicts.
Attention heads: several mixtures in parallel
Different projections let a layer combine information through more than one attention pattern.
Feed-forward layers: transforming each position
Attention moves information between positions. Feed-forward sublayers transform the features available at each one.
The residual stream: updating a running representation
Residual connections let sublayers add changes to a representation that continues through the network.
Learning
How predictions become a signal that changes the model.
Loss: a number to learn from
A prediction becomes a training signal when we score how much probability it gave the observed token.
Gradient descent: changing the weights
A gradient points uphill in loss. An optimizer uses it to take a carefully sized step in the other direction.
Backpropagation: tracing responsibility
The chain rule carries a loss signal backward through the computations that produced it.
Pretraining: learning from continuations
Large collections of examples teach a model reusable patterns before any particular assistant conversation.
Scaling laws: trends, not guarantees
Empirical curves help allocate training resources, but they do not certify the behavior of a finished assistant.
Supervised fine-tuning: demonstrations of behavior
Examples of good responses shift a pretrained model toward the behavior an application needs.
Preference learning: comparing alternatives
Sometimes people can judge which of two answers is better more easily than they can write an ideal answer.
RLHF: optimizing a learned reward
A common historical recipe learns a reward from comparisons, then uses it to improve a response policy.
DPO: learning directly from preference pairs
Direct Preference Optimization uses chosen and rejected responses to train a policy without a separate reward-model rollout loop.
Generating
From raw scores to a response, one new token at a time.
Next-token prediction: a small objective with broad demands
The model assigns probabilities to what could follow a prefix. Training rewards probability on the observed continuation.
Logits: scores before probabilities
The network ends a prediction step with one real-valued score per vocabulary entry.
Softmax: turning scores into a distribution
Exponentiate, normalize, and obtain a set of nonnegative probabilities that add to one.
Temperature: reshaping the odds
A sampling setting makes an existing distribution sharper or flatter without teaching the model anything new.
Sampling: choosing the next token
A distribution offers possibilities. A decoding rule turns those possibilities into one continuation.
Autoregression: one choice becomes the next input
Generation repeatedly predicts, selects, appends, and predicts again.
Context windows: what is available right now
The current sequence is bounded. Fitting information into it and successfully using that information are separate questions.
The KV cache: reusing the past
Causal generation can retain attention keys and values from previously processed tokens.
Systems & trust
The model is one part of the product. Understand the rest.
Base model versus assistant
The model that continues text and the product that answers your message are related, but they are not the same thing.
Prompt structure: what the model receives
A conversation becomes a structured sequence with roles, separators, instructions, and context.
In-context learning: adapting without weight updates
Instructions and examples in the input can change a model’s behavior while its learned parameters stay fixed.
Retrieval: giving the model evidence
A retrieval system finds relevant material and places it where a language model can use it.
Tool use: letting a system act
A model can propose a tool call; application code validates it, runs it, and returns the result.
Hallucinations: fluent text without support
A response can be well formed and plausible while inventing a fact, source, or event.
Evaluation: measuring the behavior you need
A useful score starts with a clear task, a representative test set, and a rule for deciding success.
Memorization: when training text comes back
A model can learn reusable patterns and also reproduce particular training sequences.
Generalization: beyond the examples seen
A learned pattern is useful when it transfers to inputs that were not simply repeated from training.
Compute: the work behind the words
Training and inference both require computation, but their memory, arithmetic, and throughput demands differ.
Latency: why a response takes time
The wait before the first token and the pace of later tokens come from different parts of the serving pipeline.
Quantization: fewer bits per number
Lower-precision representations can reduce model storage and memory traffic while introducing approximation.
Context management: choosing what stays in view
A finite input budget makes selecting, preserving, and refreshing context part of the application’s job.