What is gradient descent?
Gradient descent is the method by which a model learns from its mistakes. After the model makes predictions on a batch of training examples, the loss function measures how wrong they were. The gradient is the set of directions in which each parameter would need to move to reduce that loss fastest. Gradient descent nudges every parameter a small step in that direction, then repeats with the next batch. Over millions of steps, the loss falls and the model’s predictions improve.
The size of each step is the learning rate. The number of examples considered per step is the batch size. In practice almost all training uses mini-batch stochastic gradient descent, which estimates the gradient from a small random batch rather than the whole dataset, because it is faster and the noise helps the model avoid poor solutions. Variants such as momentum and Adam adapt the step size per parameter. Backpropagation is the technique that computes the gradient efficiently through a neural network.
Why gradient descent matters
Gradient descent is the reason training is expensive. Every step requires a forward pass to make predictions and a backward pass to compute gradients, across the entire model, for every batch. The compute bill for training a model is close to the number of gradient descent steps multiplied by the cost of each. It is also why training can fail quietly: a learning rate that is too high makes the loss bounce or explode, and one that is too low wastes days of time on graphics processing units (GPUs) creeping toward an answer.
Gradient descent in practice
An insurance company fine-tunes a language model to extract loss details from adjuster reports. The first run uses the learning rate from a blog post and the loss curve jumps around without settling. The engineer lowers the learning rate by a factor of ten, adds a short warmup period, and watches the loss decline smoothly on both the training set and the validation set. The run finishes overnight on one GPU, and the loss curve is kept with the checkpoint as evidence that training behaved.
What to watch for
- Watch the validation loss as well as the training loss, because a model whose training loss keeps falling while validation loss rises is overfitting.
- Budget training time in steps and GPU-hours, since both scale with model size and dataset size.
- Save checkpoints during long runs so a crash or a bad learning rate does not cost the whole run.