Surrogate Models for Optimization
Standing in for an expensive function
A surrogate model is a cheap-to-evaluate approximation of an expensive-to-evaluate function, trained on whatever observations of the real function have been collected so far. The entire premise of surrogate-based optimization is that evaluating the true objective — a full model training run, a physical simulation, a wind-tunnel test — is slow and costly, while evaluating a statistical model fit to prior observations is nearly instant. Instead of searching the true objective directly, you search the surrogate, and only query the real, expensive function at points the surrogate suggests are promising.
Gaussian Processes as the classical surrogate
Gaussian Processes (GPs) are the default surrogate in Bayesian optimization because they naturally provide calibrated uncertainty, not just point predictions — essential for deciding whether to exploit a region the model is confident about or explore one it isn't. A GP defines a distribution over functions; conditioning on observed data points yields a posterior mean (the prediction) and posterior variance (the uncertainty) at any new input. Their main weakness is cubic scaling in the number of observations ($O(n^3)$ for exact inference), which makes them impractical past a few thousand data points without sparse or approximate variants.
Common surrogate model families
| Surrogate | Strength | Weakness |
|---|---|---|
| Gaussian Process | Principled uncertainty estimates | Cubic scaling, struggles with high dimensions |
| Random Forest / GBM | Handles categorical & high-dim spaces well | Uncertainty estimates are cruder |
| Tree-structured Parzen Estimator (TPE) | Cheap, handles conditional spaces | Weaker uncertainty calibration than GP |
| Neural network surrogate | Scales to many observations, flexible | Needs more data to be reliable; less calibrated |
from sklearn.ensemble import RandomForestRegressor
import numpy as np
class SurrogateOptimizer:
"""Generic surrogate-driven search loop (model-agnostic)."""
def __init__(self, expensive_fn, surrogate=None):
self.expensive_fn = expensive_fn
self.surrogate = surrogate or RandomForestRegressor(n_estimators=100)
self.X, self.y = [], []
def suggest_next(self, candidates):
if len(self.X) < 3:
return candidates[np.random.randint(len(candidates))]
self.surrogate.fit(np.array(self.X), np.array(self.y))
preds = self.surrogate.predict(candidates)
return candidates[np.argmax(preds)] # pure exploitation for simplicity
def step(self, candidates):
x_next = self.suggest_next(candidates)
y_next = self.expensive_fn(x_next) # the ONE expensive call per iteration
self.X.append(x_next); self.y.append(y_next)
return x_next, y_nextSurrogates generalize past hyperparameter tuning
The fidelity-cost tradeoff
Surrogates are never free of a tradeoff between fidelity and cost — a more expressive surrogate (deeper GP kernels, larger neural networks) fits the true function more accurately but requires more data and more compute to train and query, eroding the speed advantage that justified using a surrogate at all. Multi-fidelity optimization addresses this directly by using surrogates at several levels of approximation simultaneously — a cheap, low-fidelity simulation to narrow the search broadly, and the expensive, high-fidelity true function only to confirm the most promising few candidates. This layered approach is common in both engineering optimization and modern hyperparameter tuning (e.g. combining a surrogate with Hyperband-style early stopping).
What's Next?
Choosing which point to evaluate next with a surrogate model is, at its core, a question of which unlabeled example is most worth the cost of obtaining its true value — the same question active learning asks about which data point is most worth labeling.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.