What are tokens?
Tokens are the pieces of text a language model works with. Before a prompt reaches the model, a tokenizer splits it into tokens drawn from a fixed vocabulary, typically of 30,000 to 250,000 entries. Common English words are usually one token each. Longer or rarer words are split into several pieces, so “unreimbursed” might become three tokens while “the” is one. Punctuation, spaces, numbers, and code symbols consume tokens too. The model reads the prompt as a sequence of token IDs and produces its answer one token at a time.
A useful rule of thumb for English prose is that 100 tokens covers about 75 words. Other languages, especially those that do not use the Latin alphabet, often need more tokens for the same meaning. Tables, JSON, and long numbers are expensive because nearly every digit, bracket, and separator becomes a token of its own.
Why tokens matter
Tokens are the unit in which nearly everything about a language model is measured. The context window is a token limit. Hosted APIs bill per token, with input and output tokens priced separately. Throughput is quoted in tokens per second. When a request fails because a document is too long, or a monthly invoice is higher than expected, the explanation is in the token count. Managers who can estimate tokens can estimate capacity and cost before a project starts.
Tokens in practice
A collections agency wants an assistant to read account histories before each call. A typical history is 2,000 words of notes, which is roughly 2,700 tokens. Add a 600-token system prompt and a 400-token answer, and each call consumes about 3,700 tokens. At 20,000 calls a month, the team can compute total volume, check it against the context window, and size GPU capacity or an API budget with real numbers. Trimming the notes to the most recent twelve months reduced the per-call count substantially without hurting answer quality.
What to watch for
- Different models use different tokenizers, so the same document has a different token count on each model.
- Output tokens are slower and usually cost more than input tokens, so cap answer length where a short answer will do.
- Scanned documents converted by OCR often contain stray characters that inflate token counts.
- Count the tokens in the system prompt and retrieved context, which are sent on every request and often outweigh the user’s question.