There's a subtlety that trips people up constantly: 'linear regression' doesn't mean the fitted curve is a straight line. Polynomial regression fits curves while staying linear in the parameters — y = b0 + b1*x + b2*x^2 is linear in b0, b1, b2, even though it plots as a parabola. You can still solve for the coefficients with the same closed-form matrix algebra as ordinary regression.

Non-linear least squares is for a genuinely different situation: when the parameters themselves appear inside a non-linear function, with no way to rewrite the model as a linear combination of them. Examples: exponential decay y = a * exp(-b*x), a logistic growth curve y = L / (1 + exp(-k*(x-x0))), or a Michaelis-Menten enzyme kinetics curve. In each case there is no algebraic trick that turns the equation into something solvable with (X^T X)^-1 X^T y.

Note

Some non-linear-looking relationships can be transformed into linear ones — taking the log of y = a*exp(bx) gives ln(y) = ln(a) + bx, which is linear in ln(a) and b. When that trick works, use ordinary linear regression on the transformed data; it's faster, more stable, and has a closed-form solution. Reach for true non-linear least squares only when no such transformation exists, or when you need to fit in the original (untransformed) error scale.

Why you can't just solve it directly

For linear regression, minimizing the sum of squared residuals leads to a system of linear equations with one exact solution. For non-linear models, the residual's dependence on the parameters is non-linear, so setting the derivative of the loss to zero produces a system of non-linear equations with no general closed-form solution.

Instead, non-linear least squares is solved iteratively: start from an initial guess for the parameters, repeatedly adjust them in a direction that reduces the sum of squared residuals, and stop when the improvement becomes negligible. This is fundamentally an optimization problem, much like training a neural network — except here the model usually has only a handful of interpretable parameters instead of millions of weights.

Levenberg-Marquardt: blending two strategies

The workhorse algorithm for non-linear least squares is Levenberg-Marquardt (LM). It blends two simpler optimization strategies, picking the better one adaptively at each step:

Gauss-Newton approximates the problem locally as if it were linear (using the Jacobian of the residuals) and jumps straight to the minimum of that local approximation. It converges extremely fast near the solution but can overshoot wildly, or even diverge, far from it.

Gradient descent just takes a small step downhill along the negative gradient. It's slow but reliably stable, even far from the solution.

Levenberg-Marquardt interpolates between the two using a damping parameter: far from the solution it behaves like gradient descent (safe, slow), and as it gets close it shifts toward Gauss-Newton (fast, precise). It tunes this damping automatically based on whether recent steps improved the fit, which is why LM converges reliably on most real curve-fitting problems without manual tuning.

python

Note

Unlike linear regression, non-linear least squares has no guarantee of finding the global optimum. A poor starting guess can cause the optimizer to converge to a local minimum, diverge entirely, or converge absurdly slowly. Always inspect your data to construct a sensible initial guess (e.g. estimate the asymptote and decay rate visually before fitting), and consider fitting from several different starting points to check for consistency.

Evaluating the fit

Because there's no closed-form solution, it's worth checking convergence explicitly rather than assuming the returned parameters are correct. Useful diagnostics: plot residuals against x and confirm they show no systematic pattern (a clear pattern means the chosen functional form is wrong, not just poorly fit); examine the covariance matrix returned by the optimizer to get standard errors on each parameter; and try multiple random initializations to confirm the solution is stable rather than an artifact of a lucky or unlucky starting point.

Linear vs. non-linear least squares

Aspect	Linear least squares	Non-linear least squares
Solution method	Closed-form matrix algebra	Iterative optimization (e.g. Levenberg-Marquardt)
Global optimum guaranteed?	Yes	No — depends on initial guess
Needs starting values?	No	Yes, and quality matters a lot
Typical use case	Linear trends, polynomial curves	Exponential decay, logistic growth, saturation curves
Computation cost	Fast, single step	Slower, multiple iterations

What's next

Non-linear least squares fits a specific, chosen functional form to your data. A related but different problem is classification with flexible decision boundaries rather than curve fitting — covered next in k-Nearest Neighbors, a model that makes almost no assumptions about functional form at all.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →