What is streaming?
Streaming is the practice of sending a language model’s output to the client as it is produced, a few tokens at a time, rather than holding the whole response until generation finishes. A model generates one token at a time anyway; streaming forwards each one over an open connection, usually with server-sent events or a WebSocket. The user sees the first words after the time to first token and then watches the rest arrive at the model’s tokens-per-second rate.
The total time to the last token is the same either way, but perceived latency changes. A response that takes eight seconds feels slow when the screen is blank the whole time and acceptable when words start appearing after one.
Why streaming matters
For interactive use, such as agent assist in a contact center or a chat interface, streaming is the difference between a product people use and one they abandon. It also lets the client stop generation early when the user has what they need. For background jobs such as nightly document classification, streaming adds complexity for no benefit.
What to watch for
- Every component in the path, including the AI gateway and any proxies, must pass partial responses through without buffering, or streaming quietly turns back into waiting.
- Output guardrails that need the full text cannot run until the end, so decide whether to filter chunks as they pass or hold back risky responses.
- Structured outputs and function calls are unusable until complete, so stream only the parts meant for a person.