What is speculative decoding?
Speculative decoding is an inference technique that makes a large language model generate text faster without changing what it produces. A small, fast draft model guesses the next several tokens. The large model then checks all of those guesses in a single forward pass, which costs about the same as generating one token. Every guess that matches what the large model would have produced is accepted; at the first mismatch the large model’s own token is used and drafting resumes from there.
Why speculative decoding matters
Generating tokens one at a time leaves most of a GPU idle, because each step is limited by moving the weights from memory rather than by arithmetic. Speculative decoding fills that idle capacity with verification work. When the draft model guesses well, which it does on predictable text such as boilerplate, code, and structured output, generation speed can double or more. On unpredictable text the gain shrinks and can disappear. For a contact center assistant that drafts responses while a customer waits, the difference is visible on screen. Some models ship with a built-in draft head, so no separate draft model is needed.
What to watch for
- The draft model must share the large model’s tokenizer, so the pair has to be chosen together.
- Gains depend on the workload, so measure tokens per second on real prompts rather than trusting a headline figure.
- It helps individual response speed most and matters less when the server is already saturated.