Lagrange Multipliers & Constrained Optimization
Sometimes you can't just go downhill
Plain gradient descent finds the minimum of a function by walking downhill freely. But many real problems require the solution to satisfy a constraint — a probability distribution must sum to 1, a classifier's margin must stay outside a boundary, a budget must not be exceeded. Lagrange multipliers are the classical tool for solving exactly this kind of problem: minimize (or maximize) a function subject to an equality constraint.
The geometric intuition
Suppose you want to minimize $f(x,y)$ subject to the constraint $g(x,y)=0$. Picture $f$'s contour lines (curves of equal value) on a 2D plane, and the constraint $g(x,y)=0$ as a curve threading through that plane. As you slide along the constraint curve, $f$ changes — until you reach a point where the constraint curve is tangent to one of $f$'s contour lines. At that tangent point, moving along the constraint no longer decreases $f$, so it's a candidate minimum.
Tangency between two curves means their gradient vectors are parallel — pointing in the same or exactly opposite direction. That's the entire idea behind Lagrange multipliers: find the point where $\nabla f = \lambda \nabla g$ for some scalar $\lambda$ (the Lagrange multiplier), while also satisfying $g(x,y)=0$.
The Lagrangian and a worked example
Define the Lagrangian $\mathcal{L}(x,y,\lambda) = f(x,y) - \lambda g(x,y)$. Setting its gradient (with respect to $x$, $y$, and $\lambda$) to zero recovers both the tangency condition and the constraint itself.
Worked example: minimize $f(x,y) = x^2 + y^2$ subject to $g(x,y) = x + y - 4 = 0$ (i.e. $x+y=4$). The Lagrangian is $\mathcal{L} = x^2+y^2 - \lambda(x+y-4)$. Taking partial derivatives: $\partial \mathcal{L}/\partial x = 2x - \lambda = 0 \Rightarrow x = \lambda/2$; $\partial \mathcal{L}/\partial y = 2y-\lambda=0 \Rightarrow y=\lambda/2$. So $x=y$. Substituting into the constraint: $x+x=4 \Rightarrow x=2, y=2$, and $\lambda=4$. The minimum value is $f(2,2) = 4+4=8$.
Note
From equality to inequality: KKT conditions
Most real ML constraints aren't strict equalities — SVMs require margins to be "at least" a certain size, not exactly equal to it. The Karush-Kuhn-Tucker (KKT) conditions extend Lagrange multipliers to handle inequality constraints like $h(x) \le 0$. The key addition is complementary slackness: either the constraint is exactly tight (active, $h(x)=0$) and its multiplier can be positive, or the constraint has slack ($h(x)<0$) and its multiplier must be exactly zero.
This single property is the mathematical origin of the term "support vector" in SVMs: only the data points that lie exactly on the margin boundary (where the constraint is tight) get a nonzero multiplier and therefore influence the decision boundary. Every other point — safely on the correct side with room to spare — contributes nothing to the solution.
Where this shows up: the SVM margin
A support vector machine finds the hyperplane $\vec{w}\cdot\vec{x}+b=0$ that separates two classes while maximizing the margin between them. This is posed as: minimize $\frac{1}{2}\|\vec{w}\|^2$ subject to $y_i(\vec{w}\cdot\vec{x}_i+b) \ge 1$ for every training point $i$ (each point correctly classified with at least unit margin).
Applying Lagrange multipliers (one $\alpha_i \ge 0$ per data point, following the KKT framework) turns this into a problem where the optimal $\vec{w}$ is expressed purely as a weighted combination of the support vectors: $\vec{w} = \sum_i \alpha_i y_i \vec{x}_i$, with $\alpha_i=0$ for every point that isn't a support vector. The entire "maximum margin" idea that makes SVMs work is, mechanically, a constrained optimization problem solved via Lagrange multipliers and KKT conditions.
Unconstrained vs. constrained optimization
| Aspect | Unconstrained (gradient descent) | Constrained (Lagrange / KKT) |
|---|---|---|
| Goal | Minimize $f(x)$ freely | Minimize $f(x)$ subject to $g(x)=0$ or $h(x)\le 0$ |
| Core condition | $\nabla f = 0$ | $\nabla f = \lambda \nabla g$ (tangency) plus the constraint itself |
| Inequality handling | Not applicable | KKT complementary slackness decides which constraints are active |
| Classic ML example | Plain neural network training loss | SVM margin maximization, probability-sum constraints |
What's next
Lagrange multipliers solve constrained problems exactly when the structure is tractable. When it isn't — when you only have the ability to sample rather than solve in closed form — the next lesson, Monte Carlo Methods in Machine Learning, covers how randomness itself becomes the computational tool.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.