Decision Boundaries Visualized
A decision boundary is where the model changes its mind
For a classifier, the decision boundary is the surface in feature space where predicted class membership flips from one class to another — formally, where the decision function equals the threshold that separates classes (often zero, for a model outputting a signed score, or 0.5, for one outputting a probability). Visualizing decision boundaries, usually on 2D toy datasets, is one of the most direct ways to build intuition for how different algorithms carve up feature space, because the shape of the boundary is a direct consequence of each algorithm's underlying assumptions.
Linear boundaries: logistic regression and linear SVM
Logistic regression and linear SVM both produce a boundary that's a straight line (in 2D) or hyperplane (in higher dimensions): $w \cdot x + b = 0$. This is a direct consequence of the model being linear in its parameters — there's no mechanism in either model for the boundary to curve. When the true boundary separating classes is genuinely non-linear, both will underfit no matter how the linear parameters are tuned, which is a structural (not a tuning) limitation.
import numpy as np
import matplotlib.pyplot as plt
def plot_decision_boundary(model, X, y, resolution=0.02):
x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx1, xx2 = np.meshgrid(
np.arange(x1_min, x1_max, resolution),
np.arange(x2_min, x2_max, resolution),
)
grid = np.c_[xx1.ravel(), xx2.ravel()]
Z = model.predict(grid).reshape(xx1.shape)
plt.contourf(xx1, xx2, Z, alpha=0.3, cmap="coolwarm")
plt.scatter(X[:, 0], X[:, 1], c=y, edgecolor="k", cmap="coolwarm")
plt.title(f"{model.__class__.__name__} decision boundary")
# Compare side by side:
# LogisticRegression -> straight line
# SVC(kernel="rbf") -> smooth curved boundary
# DecisionTreeClassifier -> axis-aligned rectangular regions
# KNeighborsClassifier(n_neighbors=1) -> jagged, highly irregular boundaryNon-linear and non-smooth boundaries
Kernel SVMs (with RBF or polynomial kernels), neural networks, and gradient boosting can all produce smoothly curved, flexible decision boundaries that bend to follow the actual class structure, at the cost of more hyperparameters controlling how flexible that curve is allowed to become. Decision trees produce boundaries built entirely from axis-aligned splits — rectangular regions with sharp, right-angle corners — because a tree can only ever split on 'is feature $x_i$ greater than threshold $t$,' never on a combination of features in one step; this is why trees can struggle with diagonal class boundaries unless given enough depth to approximate them with many small steps. k-NN with $k=1$ produces an extremely jagged, example-by-example boundary (a Voronoi-like partition) that follows the training data's noise almost exactly.
| Model | Boundary shape | Flexibility source |
|---|---|---|
| Logistic regression / linear SVM | Straight line / hyperplane | None — fundamentally linear |
| Kernel SVM (RBF) | Smooth curve | Kernel trick maps to higher-dim space |
| Decision tree | Axis-aligned rectangles | Recursive single-feature splits |
| k-NN (small k) | Jagged, irregular | Local majority vote among nearest points |
| Neural network | Smooth, highly flexible curve | Composed non-linear layers |
Boundary complexity is a direct visual proxy for overfitting risk
2D intuition doesn't always transfer to high dimensions
What's Next?
Among curved-boundary methods, support vector machines have a particularly well-developed theory for exactly how the boundary is chosen — not just that it curves, but why it's placed where it is relative to the data.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.