What are temperature and sampling?
Temperature and sampling are the settings that decide how a large language model turns its probability estimates into words. At each step the model scores every token in its vocabulary, and sampling is the rule for choosing one. Temperature reshapes those scores before the choice. A low temperature sharpens the distribution so the most likely token wins almost every time, while a higher temperature flattens it so less likely tokens are picked more often. A temperature of zero picks the top token every time.
Two related settings restrict the pool. Top-k keeps only the k highest-scoring tokens. Top-p, or nucleus sampling, keeps the smallest set of tokens whose combined probability reaches p. Both prevent an occasional very unlikely token at high temperature.
Why temperature and sampling matter
The settings trade consistency for variety. Extracting fields from invoices, classifying tickets, or producing structured outputs should run at low temperature so the same input gives the same answer. Drafting ten alternative product descriptions for a retailer benefits from a higher temperature, which also raises the chance of a plausible hallucination because the model is more willing to follow a less supported continuation.
What to watch for
- Even at temperature zero, output may differ slightly between runs on different hardware or batch sizes, so design for small variations.
- Record the sampling settings with every evaluation, since a model that scored well at one temperature may behave differently at another.
- Vendors scale temperature differently, so a value that works on one model does not transfer to another without testing.