What is the KV cache?
The KV cache is the working memory of a running language model. In the attention mechanism, every token is turned into a key vector and a value vector that later tokens consult. Without a cache, generating each new token would mean recomputing those vectors for the entire prompt and everything written so far. The KV cache stores them once and appends an entry for each token generated, so the model does a fixed amount of work per token regardless of how long the conversation has become. The cache lives in GPU memory next to the model weights, and for a mid-sized model a single long conversation can take several gigabytes.
Why the KV cache matters
The KV cache is usually what limits how many users a self-hosted model can serve at once. A GPU that holds the weights with 20 GB to spare can run only as many concurrent sessions as fit in that 20 GB, and one long-context request can consume most of it alone. Techniques such as paged attention and KV cache quantization raise the ceiling. A hospital planning an on-premises assistant for several hundred clinicians sizes its GPU memory by cache demand at peak as much as by model size.
What to watch for
- Ask for concurrency figures at the context lengths the application will actually use.
- Watch for requests being queued or rejected when the cache fills, which appears as rising time to first token.
- Test KV cache quantization on long conversations before relying on it.