Context windows: what is available right now
The current sequence is bounded. Fitting information into it and successfully using that information are separate questions.
In this lesson
Context is temporary input, not the entire training corpus and not unlimited memory.
A working input sequence
A context window is the sequence capacity used for a model’s current computation. Depending on the system, that sequence can include instructions, conversation turns, retrieved material, tool results, and generated tokens. The exact limit and accounting rules are model- and product-specific; this lesson uses illustrative budgets rather than advertising a current model limit.
Imagine a desk with space for twenty index cards. The cards on the desk are available for the current task; the contents of the entire library are not. The analogy has limits, but it usefully distinguishes current input from the statistical patterns already learned in the weights.
Budget in tokens, not words
Different strings can require different token counts, so a word count is only a rough proxy. Instructions and formatting can also consume space. When planning a response, distinguish the input already present from the maximum additional output that a system allows. Their relationship must be checked in the relevant model documentation.
In this site’s toy budget, suppose capacity is 100 units, existing input uses 70, and 20 are reserved for output. Ten units remain for another passage. These are deliberately small illustrative numbers. The arithmetic demonstrates budget allocation, not the behavior of a named commercial service.
A finite budget has to fit the request.
- 8 Instructions + question
- 5 Retained history
- 3 Reserved for output
Available does not mean reliably used
A passage can fit inside the window and still be difficult for a model to use. Lost in the Middle found position-dependent performance on long-context tasks in its tested models, including poorer access to information in the middle. This is a dated empirical finding, not a universal curve for every current model. Liu and colleagues, 2023.
A useful evaluation moves the same supporting fact to different positions while keeping the question and irrelevant material controlled. If performance changes, capacity alone did not describe the capability. Test extraction, comparison, and multi-step use separately because finding one sentence is not the same as synthesizing a whole document.
When the window fills
An application must handle overlong input somehow: reject it, truncate selected content, summarize, retrieve fewer passages, or use a different supported configuration. These choices are not equivalent. Removing an early instruction can change behavior; summarizing can lose a detail; rejecting preserves the error instead of silently changing the task.
The context demo makes the chosen policy visible. Change the budget and inspect what remains available. A good explanation names what is omitted and what is reserved, rather than saying simply that the model “forgot”. Context management is an application decision as well as a model constraint.
A transparent toy budget
Capacity 100 − instructions 10 − conversation 45 − retrieved passage 15 − output reserve 20 = 10 units free. Increasing the retrieved passage to 30 creates a 5-unit overflow. These units are illustrative tokens under the demo’s stated counting rule.
Fit the conversation
An illustrative word-piece budget, not any provider’s token count.
A large context window means perfect memory of everything inside it.
Capacity describes what can be supplied. Reliable access and reasoning over that content need separate testing.
Take it with you
- Context is temporary input distinct from learned weights.
- Budgeting depends on the actual tokenizer and system rules.
- Long-context capacity is not a guarantee of effective use.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
The KV cache: reusing the past
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Lost in the Middle: How Language Models Use Long Contexts ↗Nelson F. Liu et al. · 2023-11-20 (v3) · checked 13 Nov 2026