When you run gradient descent, you're walking downhill on a loss surface defined over every parameter in the model. For some models that surface has a single, predictable valley. For deep neural networks it looks more like a vast, bumpy mountain range with countless valleys, ridges, and flat plateaus. The distinction between these two kinds of surfaces — convex and non-convex — explains both why classical ML models (like linear regression) train so reliably, and why deep learning training is a much messier, more empirical process.

Convex functions: one valley, guaranteed

A function is convex if the line segment between any two points on its graph lies above (or on) the graph itself: $f(\lambda x_1 + (1-\lambda)x_2) \le \lambda f(x_1) + (1-\lambda)f(x_2)$ for $\lambda \in [0,1]$. In practice this means: no local minima other than the global minimum, and gradient descent with a reasonable learning rate is mathematically guaranteed to converge there.

$f(x) = x^2$ is the textbook convex function. Check two points: $x_1=-2, x_2=4$, midpoint $\lambda=0.5$ gives $x=1$. $f(1)=1$. The chord value is $0.5f(-2)+0.5f(4) = 0.5(4)+0.5(16) = 10$. Since $1 \le 10$, the convexity condition holds here, as it does everywhere for $x^2$ — it's a single smooth bowl.

python

Non-convex functions: many valleys, no guarantees

Neural network loss functions are non-convex because they compose many nonlinear activation functions and matrix multiplications across layers — stacking these transformations produces a loss surface with many local minima (points that are locally the lowest, but not globally), saddle points (flat in some directions, sloping in others), and ridges.

$f(x) = x^4 - 4x^2$ is a simple non-convex example: it has two separate local (and in this case global) minima at $x=\pm\sqrt{2}$, with a local maximum at $x=0$ in between. $f(\sqrt{2}) = 4 - 8 = -4$ and $f(-\sqrt{2}) = -4$ — two equally good basins with a hill separating them. Gradient descent starting near $x=0.1$ slides toward one minimum; starting near $x=-0.1$ slides toward the other. Which one you reach depends entirely on where you started.

python

Local minima vs. saddle points: different kinds of "stuck"

A local minimum is a point where the function is lowest in every direction nearby — the gradient is zero and the surface curves upward on all sides. A saddle point is also a point where the gradient is zero, but the surface curves upward in some directions and downward in others — like the middle of a horse saddle.

In high-dimensional spaces (the regime deep networks actually train in), saddle points vastly outnumber true local minima. This is a consequence of random matrix theory: for a critical point to be a local minimum, the surface must curve upward in every one of the thousands or millions of parameter directions simultaneously — an exponentially unlikely coincidence. For it to be a saddle, it only needs to curve downward in at least one direction, which is comparatively easy to satisfy.

Note

Because saddle points dominate over bad local minima in high dimensions, and because saddle points have at least one downhill escape direction, stochastic gradient descent's inherent noise (from mini-batching) tends to kick training out of saddle points rather than getting permanently trapped. Empirically, most local minima found in large networks turn out to have loss values close to the global minimum anyway — the loss landscape is "bumpy" but the bumps aren't catastrophic.

Note

Even though saddle points are usually escapable, the region immediately around one can be nearly flat, causing gradients to shrink toward zero and training to stall for many steps before an escape direction is found. This is one practical reason optimizers like Adam (with adaptive per-parameter step sizes) and momentum-based methods often train faster than plain SGD — they push through these flat regions more aggressively.

Convex vs. non-convex optimization

Property	Convex	Non-convex (deep learning)
Number of minima	Exactly one (the global minimum)	Many local minima and saddle points
Convergence guarantee	Gradient descent provably reaches the global minimum	No guarantee; result depends on initialization and optimizer
Typical example	Linear/logistic regression loss	Neural network loss (any hidden layer with nonlinear activation)
Main obstacle	Choosing a learning rate that doesn't overshoot	Saddle points, flat regions, and local minima of varying quality

What's next

Non-convexity means unconstrained gradient descent, but real ML problems often come with explicit constraints — like requiring a margin in SVMs or a probability distribution to sum to one. The next lesson — Lagrange Multipliers & Constrained Optimization — covers the classical technique for optimizing a function subject to exactly those kinds of constraints.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →