What is batch size?
Batch size is how many training examples the model sees between one parameter update and the next. A batch size of thirty-two means the model predicts on thirty-two examples, averages the error, and takes one gradient descent step. It is set before training, which makes it a hyperparameter, and it interacts closely with the learning rate: larger batches give a smoother gradient estimate and usually tolerate a larger step.
The main constraint is the memory of the graphics processing unit (GPU). Every example in a batch needs memory for its activations during the forward and backward passes, so the largest batch that fits is set by the model size and the hardware. Larger batches use the GPU more efficiently and finish an epoch in fewer steps. Smaller batches add noise to each update, which can help the model generalize, and they fit on smaller hardware. When the desired batch does not fit, gradient accumulation runs several small batches and applies their combined update as if they were one.
Batch size in practice
A regional insurer fine-tunes a seven-billion-parameter language model with a LoRA adapter on a single GPU with twenty-four gigabytes of memory. A batch of eight claim summaries runs out of memory. The engineer drops to a batch of two and accumulates gradients over four batches, which behaves like a batch of eight while fitting on the card. Training takes longer in wall-clock time but completes on hardware the company already owns.