What is rate limiting?
Rate limiting is the practice of bounding how much a caller may ask of an AI service in a window of time. Limits are usually expressed as requests per minute, tokens per minute, or concurrent requests, and they can apply per API key, per user, per application, or to an entire organization. When a caller exceeds the limit, the service refuses the extra requests with a standard error, often HTTP status 429, and tells the client how long to wait.
Why rate limiting matters
Hosted providers impose rate limits to share their capacity fairly, and companies running their own models set limits for the same reason. A single GPU has a fixed amount of KV cache and a fixed throughput in tokens per second. One batch job that fires 5,000 document summaries at once can fill the queue and leave every interactive user staring at a spinner. Per-team limits, enforced at an AI gateway, keep a logistics company’s overnight manifest processing from crowding out the customer service desk. Token limits are more accurate than request limits, since one request with a 100-page attachment costs more than a hundred short questions.
What to watch for
- Build retry with backoff into every client from the start, because the limit error will arrive eventually.
- Set limits per team or application rather than one shared pool, so a noisy job cannot starve the rest.
- Give interactive traffic priority over batch traffic in the gateway or inference server queue.