Context Window Explained

What it is

The context window is the maximum amount of text, measured in tokens, that a language model can take into account when generating its next token. It covers everything: your prompt, any documents you pasted in, the conversation so far, and the model's own output as it generates. When the total exceeds the window, the oldest content falls off and the model can no longer see it.

Think of it as the model's working memory. A person doing a task can keep the relevant materials on the desk; anything filed away in another room might as well not exist. The context window is the desk. Retrieval systems, long documents, and extended conversations all compete for space on it.

Measured in tokens, not words

Window sizes are quoted in tokens, and tokens are not words. For English, a token averages about three quarters of a word, so a 128,000-token window holds roughly 96,000 English words. But tokenization efficiency varies wildly by language and content type. Code, with its whitespace and symbols, uses more tokens per character than prose. Some languages need two or three times as many tokens as English for the same meaning. A window size that sounds generous in English can feel cramped in another language, and cost comparisons across models are misleading when their tokenizers differ.

How windows grew

Early models had tiny windows. GPT-3, released in 2020, could consider 2,048 tokens at a time, about three pages of text. Anything longer had to be summarized or chunked, and the model forgot the beginning of long documents.

Growth since then has been rapid. By the mid-2020s, leading models advertised windows of hundreds of thousands of tokens, and some reached into the millions: entire codebases, book-length documents, hours of transcripts in a single prompt. This expansion came from a mix of architectural tweaks, more efficient attention variants, and simply training on longer sequences so the model learned to use the space.

Why windows are limited at all

The limit comes from the attention mechanism. Every token's representation is computed by comparing it against every other token, so the computation grows with the square of the sequence length. Doubling the window quadruples the attention cost. Memory grows too: during generation, the model caches the keys and values for all previous tokens, and that cache grows linearly with length. A million-token window is not just a bigger number; it is a serious engineering undertaking in memory and compute.

Longer training sequences also cost more. A model only learns to use a long window if it trains on long sequences, and those are expensive. So window size reflects real tradeoffs, not arbitrary product decisions.

Long context is not perfect recall

A common mistake is treating a large window as a guarantee that the model will use all of it well. Research has shown a "lost in the middle" effect: models pay most attention to the start and end of long inputs and underweight the middle. A fact buried in the center of a 200,000-token document is more likely to be missed than the same fact placed near the beginning.

This has practical consequences. Dumping an entire database into the prompt is usually worse than retrieving the relevant passages first. The model's attention is a finite resource even within a large window, and irrelevant content competes with relevant content for it. Retrieval-augmented generation exists partly for this reason: select the important parts, then spend the window on those.

Trained length versus usable length

A model only learns to handle sequences as long as the ones it trained on. Positional encodings, the mechanism that tells the transformer where each token sits, are learned or defined over the training range, and behavior degrades past it. A model trained on 4,000-token sequences will not magically use a 100,000-token window well just because the architecture allows it. Extending the usable window requires additional training on longer sequences or specialized techniques that stretch the positional encodings, and even then the quality at extreme lengths usually trails the quality at trained lengths. Advertised window size is a capacity number. Effective window size, the range over which the model actually reasons well, is the number that matters.

Using the window well

A few guidelines follow from how the mechanism works. Put the most important instructions and context where the model attends best, typically restating key instructions near the end of the prompt as well as the start. Keep the signal-to-noise ratio high: every irrelevant paragraph dilutes attention on the relevant ones. For conversations, remember that the window is shared between history and new output, so very long chats eventually push the earliest messages out, and the model will forget things you told it an hour ago without any warning. Summarizing the conversation so far and starting fresh is the standard workaround.

The window is also where output lives. A model generating a long response consumes its own window as it writes. Prompts that leave no room for the answer produce truncated output, not an error, which is a common source of confusion.