What is token-based pricing?
Token-based pricing is how most hosted language model services charge. Every request is metered by its input tokens (the system prompt, conversation history, and any attached documents) and by the output tokens the model generates. Each category has its own rate, usually quoted per million tokens, and output tokens cost more than input tokens because generating them takes more compute. Many providers add a reduced rate for cached input tokens, where a repeated prompt prefix was processed once and reused.
Why token-based pricing matters
The cost of a feature is set by its design more than by how often it is used. An assistant that sends a 5,000-token system prompt and a 10-page retrieved context with every one-line question pays mostly for context. A long conversation grows more expensive with every turn because the whole history is re-sent, and multi-step agents multiply this, since each tool call is another round trip with the full context attached. Teams lower the bill by trimming prompts, ordering them for cache hits, routing simple tasks to smaller models, and capping output length. The alternative is capacity-based pricing, where GPU time or owned hardware costs the same however many tokens flow through it.
What to watch for
- Track token usage per application and per team at the gateway, because the invoice arrives as one number.
- Count the hidden tokens: retries, tool definitions, and the reasoning tokens some models generate before answering.
- Re-estimate cost whenever a prompt, model, or context strategy changes.