What is regression performance?
Regression performance is how closely a model’s numeric predictions match the actual values on test data. Unlike a classifier, a regression model is rarely exactly right, so the measures describe the size and pattern of the errors. Each is computed from the residuals, which are the differences between predicted and actual values across the test set.
| Metric | What it measures | Characteristic |
|---|---|---|
| Mean absolute error (MAE) | Average size of the error in the original units | Easy to explain and treats all errors alike |
| Root mean square error (RMSE) | Square root of the average squared error | Punishes large errors more heavily than small ones |
| Mean absolute percent error (MAPE) | Average error as a share of the actual value | Comparable across items of different scale, but breaks near zero |
| R-squared | Share of the variation in the outcome the model explains | Scale-free, but can be high while predictions are still useless |
Why regression performance matters
The choice of measure reflects which errors hurt. A utility forecasting hourly electricity demand cares most about the rare large misses that force expensive spot purchases, so RMSE is the right objective. A distributor forecasting weekly demand across thousands of stock items wants a measure that compares a fast-moving item with a slow one, so MAPE or a scaled error is used, with care for items that sell nothing most weeks. Reporting the error in the business’s own units, such as hours, cases, or kilowatt-hours, lets a manager judge whether the model is good enough without translating a statistic.
Regression performance in practice
A machine shop builds a model to estimate the labor hours a custom job will take. The baseline is the estimator’s historical error. The model’s MAE on held-out jobs is lower than the estimator’s, but its RMSE is higher, which means it makes a few very large misses on unusual jobs. The shop deploys the model for routine jobs and routes anything outside the training range to the estimator. Both figures appear on the monthly model monitoring report.
What to watch for
- Compare against a baseline model, such as repeating last period’s value, because a plausible error figure can still be worse than no model.
- Plot the residuals against the prediction, since a model that is accurate on average may be wrong in a consistent direction at the extremes.
- Measure performance on a time period after the training data, because random splits flatter forecasting models.
- Ask for the distribution of errors as well as the average, since a single number hides the worst cases.