What is prompt caching?
Prompt caching is the reuse of computation for a part of a prompt that repeats across requests. As a model reads a prompt, it builds KV cache entries for every token. If the next request starts with the same tokens, the inference server keeps those entries and begins work at the point where the new request diverges. The repeated part, called the prefix, is typically a system prompt, a tool catalog, or a long document. The match must be exact from the first token, so one changed word early in the prompt defeats the cache.
Why prompt caching matters
An insurance claims assistant might send 4,000 tokens of policy rules with every 100-token question, and without caching the server re-reads those 4,000 tokens every time, which drives up time to first token and, on a hosted API, the bill. With caching, the fixed part is processed once and each request pays only for the new tokens. Hosted providers charge a reduced rate for cached input tokens; self-hosted inference servers cache automatically within GPU memory limits. Stable content therefore belongs at the front of the prompt and variable content, such as the user’s question, at the end.
What to watch for
- Order the prompt so the unchanging parts come first; a timestamp or user name near the top will break every cache hit.
- Caches expire, so a prefix used once an hour may never be found.
- Caching never changes the answer, so it is safe to enable without re-testing quality.