What is a loss function?
A loss function is the mathematical measure of how wrong a model’s predictions are on the training data. It takes the model’s outputs and the true labels and returns one number, where lower means better. Training is the process of adjusting the model’s parameters to make that number as small as possible, which is what gradient descent does step by step. The loss function therefore defines what “good” means to the model.
Different tasks use different losses. Regression models that predict a quantity usually use mean squared error, which punishes large misses much more than small ones, or mean absolute error, which treats all misses in proportion. Classification models use cross-entropy, which punishes confident wrong answers heavily. Language models are trained with cross-entropy over the next token. Specialized losses exist for ranking, for detecting objects in images, and for comparing embeddings.
Why the loss function matters
The loss function is where business priorities enter the mathematics. If a missed fraud costs far more than a false alarm, a loss that weights false negatives more heavily will produce a model that behaves accordingly. If forecasting too little stock costs a lost sale while forecasting too much costs only storage, an asymmetric loss encodes that preference directly. A model trained with a default loss will optimize for an average the business may not care about.
Loss function in practice
A distributor forecasts weekly demand for twelve thousand items. The first model uses mean squared error and is accurate on average, but the planners complain that it under-forecasts promotional weeks, causing stockouts on the items that matter most. The team switches to a quantile loss that penalizes under-forecasting more than over-forecasting for high-margin items. Average error rises slightly, stockouts fall, and the planners trust the forecast enough to stop overriding it.
What to watch for
- Ask what loss a model was trained with and whether it reflects the real cost of each kind of error.
- The loss used for training and the metric used for evaluation are often different, so check both.
- A loss that is very sensitive to outliers can let a few bad records dominate training.
- Changing the loss changes the model’s behavior everywhere, so re-run the full evaluation afterward.