Context windows, explained
You can think of it as the model's available working space. The current request shares that space with other instructions and information supplied by the application.
A context window is the amount of text, measured in tokens, that a language model can consider at one time when producing a response.
More than the visible prompt counts
Your message uses part of the context window. Earlier conversation, system instructions, uploaded documents, and retrieved passages may use the rest.
The window is measured in tokens rather than words. Different models may split the same sentence into tokens differently, so a word count is only a rough guide.
Long inputs force a choice
When the available material exceeds the limit, the application must remove or summarize something. That can make an older detail unavailable to the next response.
A larger window can hold more material, but relevance still matters. A focused set of current instructions and useful source passages is easier to inspect than a pile of unrelated text.
Context window terms
- Token
- A unit of text that may be a word, part of a word, or punctuation.
- Context length
- Another name for the maximum amount of text a model can consider at once.
- Truncation
- Removing part of the input so the remaining material fits within the limit.
What uses the available space
Your request
The current question and any documents you provide take up context.
Earlier messages
Conversation history can remain available only while it fits.
Hidden instructions
System rules and retrieved information can also use part of the window.
Questions about context windows
Not exactly. It is the material available for the current response. An application may store other information outside the window and add it later.
The application may split, summarize, or remove parts of it before sending material to the model.
The earlier message may no longer fit in the active context, or the application may have summarized the conversation and left that detail out.
Related glossary terms
Give the AI the context that matters
Caffeine apps run on the Internet Computer, a public network built for this.