What is continuous batching?
Continuous batching is the scheduling method modern inference servers use to serve many language model requests on one GPU at the same time. A GPU processes a batch of requests together, one token step at a time. In the older static approach, the server waited for a group of requests to arrive, ran them all until the longest one finished, and only then accepted new work. Continuous batching instead checks the batch at every token step: a request that has finished leaves immediately, and a waiting request joins in its place.
Why continuous batching matters
Throughput on a self-hosted model is largely a scheduling problem. Generating one token for one user and generating one token each for forty users cost nearly the same GPU time, because both are limited by reading the model weights from memory. Continuous batching exploits that by keeping as many requests in flight as the KV cache allows. A distributor running a nightly job that classifies thousands of supplier emails alongside a daytime assistant for the sales desk gets far more total work from a single GPU, and interactive users are not stuck behind the batch job.
What to watch for
- Throughput rises with batch size but per-request speed falls slightly, so set limits that protect interactive latency.
- Capacity is bounded by KV cache memory, which decides how many requests can be in the batch at once.
- Maximum batch size and queue priority are tuning decisions, so review them under realistic load.