What is time to first token (latency)?
Time to first token (TTFT) is the time from the moment an application sends a prompt to the moment the first token of the answer arrives. It covers network transit, any queue wait at the server, and the prefill phase, in which the model reads the entire prompt and fills the KV cache before it can produce anything. Prefill cost grows with prompt length, so a request carrying a 50-page document waits far longer for its first token than a one-line question does.
Why time to first token matters
People judge responsiveness by the first sign of life. A field technician asking a maintenance assistant about a fault code will tolerate an answer that streams in over several seconds if it starts within one. TTFT is also the first symptom of an overloaded server: when the KV cache is full and requests queue, the wait for the first token stretches while generation speed looks normal. Long system prompts and large retrieved contexts are the most common self-inflicted causes. Prompt caching, which reuses the prefill of a repeated prefix, is the usual fix for them.
What to watch for
- Measure TTFT separately from total response time and set a target for each.
- Track it at the busiest hour, because queueing makes it degrade before anything else does.
- Trim or cache the fixed part of every prompt, since prefill time scales with the tokens sent.
- Use streaming so users see the first token as soon as it exists.