What is long context?
Long context describes language models whose context window can hold hundreds of thousands of tokens, and the practice of using that room to feed the model entire documents, codebases, or transcripts in one request. Instead of chunking a 300-page procurement file and retrieving the pieces that seem relevant, an application can supply the whole file and ask the question directly. The model sees every clause, every exhibit, and the relationships between them.
Why long context matters
Long context removes a design step and a failure mode. Retrieval can miss the passage that matters, and chunking can split a table from its heading. Supplying the whole document avoids both problems, which is valuable for tasks such as auditing a loan file, reconciling a contract against its amendments, or reviewing a year of board minutes for a specific decision. The trade-offs are cost and speed. Each request processes every token, GPU memory for the KV cache grows with the prompt, and models still pay less attention to material in the middle of a long prompt than at its ends.
What to watch for
- A long window does not guarantee accurate recall across it, so test with questions whose answers sit deep inside the document.
- Prompt caching can offset the cost when the same long document is queried repeatedly.
- Retrieval remains the better choice when the corpus is far larger than any window, such as a whole document management system.