What are hyperparameters?
Hyperparameters are the configuration values a person sets before training a model, as opposed to the parameters the model learns from data. The learning rate, the batch size, the number of epochs, the depth of a decision tree, the number of trees in a random forest, the strength of regularization, and the rank of a LoRA adapter are all hyperparameters. They shape the training process and the model’s capacity, and the right values depend on the dataset.
Nobody can calculate the best hyperparameters in advance. They are found by trying candidates, training a model with each, and comparing results on validation data that was not used for training. Grid search tries every combination from a list; random search samples combinations; Bayesian optimization uses earlier results to pick promising candidates. Automated machine learning tools wrap this search so it runs without hand-holding.
Why hyperparameters matter
Two teams using the same algorithm on the same data can get very different results because of hyperparameters alone. A tree model that is too deep memorizes the training data and fails on new records, which is overfitting; one that is too shallow misses real patterns. The search also drives cost, because each candidate is a full training run. For a large neural network, that can mean the difference between a day and a month of time on a graphics processing unit (GPU).
Hyperparameters in practice
A wholesale distributor builds a demand forecasting model with gradient-boosted decision trees. The data scientist fixes most settings at sensible defaults and searches over the three that matter most for this data: the number of trees, the maximum tree depth, and the learning rate. Each candidate is scored by cross-validation on the past two years of orders. The winning configuration is recorded with the model in the model registry so that the result can be reproduced when the model is retrained next quarter.
What to watch for
- Tune on validation data and report final results on a separate test set, or the reported accuracy will be optimistic.
- Start with published defaults and change a few settings at a time, because most hyperparameters barely matter for a given problem.
- Record every setting alongside the trained model, since a model that cannot be reproduced cannot be maintained.
- Ask vendors how their models were tuned and on what data, because a model tuned on one customer’s data may not suit yours.