What is grid search?
Grid search is the simplest systematic way to choose hyperparameters: list a few candidate values for each setting, train a model for every combination, and keep the one that scores best on validation data. With three settings and four values each, the grid has sixty-four points. Combined with five-fold cross-validation, that is three hundred and twenty training runs, all of which can run in parallel.
The method is exhaustive within the grid, which is both its strength and its cost. It never misses a combination that was listed, and it never finds one that was not. The number of runs multiplies with every setting added, so grid search suits models that train in seconds or minutes and a handful of settings. Random search, which samples combinations instead of enumerating them, usually finds a result as good with far fewer runs when some settings matter little. Bayesian optimization goes further by using earlier results to choose the next candidate.
Grid search in practice
A hospital’s finance team builds a model to predict which insurance claims will be denied. The model is a gradient-boosted tree ensemble that trains in under a minute. The analyst grid-searches tree depth, number of trees, and learning rate over four values each, runs the full grid overnight on a laptop, and picks the configuration with the best recall on denied claims. For a neural network that takes hours per run, the same team would use random search or a Bayesian tuner instead.