What is a context window?
The context window is the amount of text, measured in tokens, that a language model can consider in a single request. Everything the model needs must fit inside it: the system prompt, prior turns of the conversation, documents the application has retrieved, tool results, and the answer the model is about to write. Anything outside the window does not exist as far as the model is concerned.
Window sizes have grown from a few thousand tokens to hundreds of thousands or more, enough to hold a long contract or a year of meeting minutes. Larger windows cost more to use. Attention compares every token with every other token, and the KV cache that stores intermediate results grows with the number of tokens, so a long prompt takes longer to process and occupies more GPU memory.
Why the context window matters
The window sets the shape of an application. If the window is small, the application must select what to include, usually through retrieval-augmented generation (RAG) and chunking, which bring in only the passages relevant to the question. If the window is large, whole documents can be supplied directly, which is simpler and often more accurate, but slower and more expensive per request. The window also bounds conversation length. A long support chat eventually fills it, and the application must summarize or drop earlier turns.
Quality does not stay flat across the window. Models attend well to material at the start and end of a long prompt and less well to material buried in the middle. A model that advertises a huge window may still miss a clause on page 80 of a 120-page lease.
Context window in practice
A commercial insurance broker builds an assistant that answers questions about a client’s policy bundle. Early tests loaded every policy document into the prompt, which fit the window but produced slow, costly answers and occasional misses on endorsements buried in the middle. The team switched to retrieving the relevant sections, with the full document available on request. Accuracy on the test set improved and response time fell.
What to watch for
- The advertised window covers input and output together, so reserve room for the answer.
- Test accuracy at the prompt lengths the application will actually use, because quality drops before the limit is reached.
- Long prompts raise time to first token, which users notice more than total generation time.
- A self-hosted model’s usable window is limited by GPU memory as well as by the model’s design.