Grid Search vs Random Search vs Bayesian Optimization
Three ways to search a hyperparameter space
Grid search, random search, and Bayesian optimization answer the same question — which hyperparameter configuration to try next — with increasingly informed strategies. Grid search enumerates every combination on a predefined discrete grid. Random search samples configurations independently at random from specified distributions. Bayesian optimization uses a probabilistic model of past results to choose the next configuration deliberately. The right choice depends heavily on dimensionality, evaluation cost, and how much of the search budget you can afford to spend.
Grid search: exhaustive but wasteful in high dimensions
Grid search's appeal is simplicity and full coverage of the chosen grid — if the optimum lies exactly on a grid point, grid search will find it. Its fatal weakness is the curse of dimensionality: evaluating even 5 values per hyperparameter across 5 hyperparameters requires $5^5 = 3125$ evaluations. Worse, grid search spends equal effort on every dimension even when some hyperparameters barely affect the objective — a well-known empirical finding (Bergstra & Bengio, 2012) is that in most real hyperparameter spaces, only a few dimensions actually matter, and grid search wastes the majority of its budget varying the ones that don't.
Random search: a surprisingly strong baseline
Random search samples configurations independently, which sounds naive but has a real advantage: because it doesn't waste a fixed number of trials per dimension, it covers the *important* dimensions more densely whenever most dimensions are unimportant — exactly the regime Bergstra & Bengio found common in practice. For the same evaluation budget, random search frequently matches or beats grid search, and it degrades much more gracefully as dimensionality increases. It's also trivially parallelizable (every sample is independent) and requires no sequential bottleneck, unlike Bayesian optimization.
Grid vs random vs Bayesian optimization
| Property | Grid search | Random search | Bayesian optimization |
|---|---|---|---|
| Scales with dimensions | Poorly (exponential) | Reasonably well | Moderate (GP cost grows with trials) |
| Uses past results | No | No | Yes — explicitly models the objective |
| Parallelizable | Fully | Fully | Partially (sequential by default) |
| Best regime | Very few hyperparameters, coarse tuning | Many trials available, cheap evaluations | Few, expensive evaluations |
| Implementation complexity | Trivial | Trivial | Moderate-to-high |
import itertools, random
def grid_search(space: dict[str, list]):
keys = list(space)
for values in itertools.product(*space.values()):
yield dict(zip(keys, values))
def random_search(space: dict[str, tuple], n_trials: int):
for _ in range(n_trials):
yield {k: random.uniform(*v) for k, v in space.items()}
# Grid: 5 values x 5 hyperparameters = 3125 evaluations, fixed and exhaustive.
# Random: choose n_trials directly, e.g. 100 — budget decoupled from dimensionality.
grid_space = {f"param_{i}": [0.1, 0.3, 0.5, 0.7, 0.9] for i in range(5)}
random_space = {f"param_{i}": (0.0, 1.0) for i in range(5)}
print(len(list(grid_search(grid_space)))) # 3125
print(len(list(random_search(random_space, 100)))) # 100, budget set directlyWhen Bayesian optimization's overhead isn't worth it
A practical decision rule
A reasonable default: use random search as the baseline whenever you have more than two or three hyperparameters to tune, since it near-always beats grid search for the same budget. Reach for Bayesian optimization (or a multi-fidelity variant like BOHB) specifically when evaluations are expensive enough that squeezing more signal out of each one — by conditioning the next choice on everything learned so far — clearly outweighs the cost of the sequential, model-fitting overhead it introduces.
What's Next?
The surrogate model underlying Bayesian optimization isn't limited to Gaussian Processes, and understanding the broader family of surrogate modeling techniques used in optimization opens the door to applying the same ideas well beyond hyperparameter tuning.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.