What is tokenization?
Tokenization is the step that splits text into the units, called tokens, that a language model reads and produces. A token can be a whole word, part of a word, a punctuation mark, or a piece of a number. Common English words are usually one token each; rarer words, technical terms, and non-English text are split into several. A rough rule for English prose is that a thousand tokens correspond to about 750 words. Each token is mapped to a number from the model’s fixed vocabulary, and those numbers are what the model actually processes.
The term has a second meaning in data security, where tokenization replaces a sensitive value such as a card number with a substitute token. The two uses are unrelated.
Why tokenization matters
Tokens are the unit of everything a business pays for and plans around with a large language model. Token-based pricing bills per token in and out. The context window is measured in tokens, so a contract that runs to a hundred pages may not fit. Throughput is measured in tokens per second. A document in German or in program code consumes more tokens than the same content in English, which changes both cost and capacity planning. Tokenization also explains some odd model behavior. A model that struggles to count the letters in a word or to do arithmetic on long numbers is working on tokens that do not line up with letters or digits.