What is a batch?
A batch is a set of examples that a model processes at the same time. During training, the model makes predictions for the whole batch, the loss function scores them together, and gradient descent applies one update to the parameters based on the average. The training data is divided into batches, and one pass through all of them is an epoch. The number of examples per batch is the batch size.
The word has a second meaning in deployment. Batch inference means running a trained model over a large set of records at once, often on a schedule, as opposed to real-time inference that answers requests one at a time as they arrive. Both senses rely on the same fact: a graphics processing unit (GPU) does the same arithmetic on many examples at nearly the cost of doing it on one, so grouping work is far more efficient than handling items individually.
Batch in practice
A collections agency scores its entire portfolio of two million accounts every night to set the next day’s call priorities. The scoring job feeds the model batches of several thousand accounts at a time and finishes in minutes. The same model, if called once per account as agents opened them, would take far longer in total and leave the GPU idle between calls. Language model inference servers apply the same idea with continuous batching, which merges requests from many users into shared batches as they arrive.