Hyperparameter Optimization with Bayesian Methods
Why hyperparameter tuning is a search problem
Learning rate, weight decay, batch size, dropout probability, number of layers — these hyperparameters shape how well a model trains, but evaluating any single configuration means actually training the model, which can take hours to weeks. This makes hyperparameter tuning an expensive black-box optimization problem: you can query the objective function (validation performance) for a given input (hyperparameter configuration), but each query is costly, and you have no gradient information about how the objective responds to the hyperparameters. Bayesian optimization is designed exactly for this regime — few, expensive evaluations, no gradients.
The surrogate model and the acquisition function
Bayesian optimization maintains a probabilistic surrogate model — most commonly a Gaussian Process (GP) — that approximates the true (expensive) objective function based on the configurations evaluated so far. Crucially, a GP gives not just a predicted value at any untried configuration but also an *uncertainty estimate*. An acquisition function then uses both the predicted mean and the uncertainty to decide which configuration to try next, explicitly balancing exploitation (try configurations predicted to be good) against exploration (try configurations where uncertainty is high and a pleasant surprise is possible). Expected Improvement (EI) and Upper Confidence Bound (UCB) are the two most common acquisition functions.
from sklearn.gaussian_process import GaussianProcessRegressor
import numpy as np
def expected_improvement(mu, sigma, best_so_far, xi=0.01):
from scipy.stats import norm
improvement = mu - best_so_far - xi
z = improvement / (sigma + 1e-9)
return improvement * norm.cdf(z) + sigma * norm.pdf(z)
def bayes_opt_step(X_observed, y_observed, candidate_grid):
gp = GaussianProcessRegressor()
gp.fit(X_observed, y_observed)
mu, sigma = gp.predict(candidate_grid, return_std=True)
ei = expected_improvement(mu, sigma, best_so_far=y_observed.max())
next_idx = np.argmax(ei)
return candidate_grid[next_idx] # hyperparameter config to evaluate nextWhy this beats blind search when evaluations are expensive
Practical limitations
Gaussian Processes scale poorly with the number of observations — exact GP inference is $O(n^3)$ in the number of evaluated points, which caps practical use at a few hundred to low thousands of trials without approximations. GPs also handle high-dimensional or purely categorical/conditional search spaces (e.g. 'number of layers' changing which other hyperparameters even apply) awkwardly compared to tree-based surrogates. This has pushed much production tooling (Optuna, Hyperopt) toward Tree-structured Parzen Estimators (TPE) as the surrogate instead of a GP — cheaper, more naturally handling conditional and categorical spaces, at some cost to the quality of the uncertainty estimate that drives exploration.
Combining with early stopping
Pure Bayesian optimization still evaluates each candidate to a fixed training budget, which is wasteful if a configuration is clearly bad after just a few epochs. Multi-fidelity methods like Hyperband and BOHB (Bayesian Optimization and Hyperband) combine the sequential, informed search of Bayesian optimization with aggressive early stopping of unpromising runs — allocate a small budget to many configurations, then only extend the budget for the survivors. This hybrid approach is now the practical default in most hyperparameter tuning libraries used for deep learning, since GPU time saved on early-stopped bad runs usually outweighs the loss in Bayesian surrogate accuracy from noisier, shorter observations.
What's Next?
Bayesian optimization is one point on a spectrum of hyperparameter search strategies, and understanding exactly where it outperforms — and where simpler methods like random search are nearly as good for a fraction of the complexity — is worth examining directly through a side-by-side comparison.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.