What is the learning rate?
The learning rate is a small number that scales each update gradient descent makes to a model’s parameters. The gradient says which direction to move each parameter, and the learning rate says how far. It is the single most consequential hyperparameter in training a neural network. Too high, and the loss oscillates or shoots to infinity because each step overshoots. Too low, and training creeps along for days without reaching a good solution, or settles in a poor one.
Modern training rarely uses one fixed value. A schedule starts with a short warmup at a tiny rate, rises to a peak, and then decays toward zero as training ends, so early steps are cautious and late steps refine. Fine-tuning a pre-trained model uses a much smaller learning rate than pre-training did, often ten to a hundred times smaller, so that the new data adjusts the model without erasing what it already knows.
Learning rate in practice
A legal services firm fine-tunes a model to classify incoming matters by practice area. The first run copies the learning rate from a pre-training recipe and the model forgets how to write coherent sentences. The second run lowers the rate by a factor of twenty, adds warmup, and produces a model that classifies well and still answers general questions. The team records the schedule with the checkpoint, so the next retraining starts from a known-good setting instead of a guess.