Polynomial Regression

A linear model that fits curves

Linear regression requires the model to be linear in its parameters — but nothing stops you from feeding it non-linear features. Polynomial regression does exactly that: instead of fitting y = b0 + b1*x, it fits

y = b0 + b1*x + b2*x^2 + b3*x^3 + ... + bd*x^d

Each power of x is treated as a separate feature, and ordinary least squares solves for the coefficients exactly as before. The model is still linear in b0...bd, so it keeps all of linear regression's nice mathematical properties — closed-form solution, no iterative optimization needed — while the resulting prediction curve bends and flexes with the data.

Choosing the degree

The polynomial degree d is a hyperparameter that controls model flexibility directly. Degree 1 is ordinary linear regression. Degree 2 can capture a single curve (one bend). Higher degrees can capture more complex wiggles — but the number of possible wiggles grows with the degree, and so does the risk of fitting noise instead of signal.

There's no formula for the 'right' degree — it depends on the true underlying relationship, which you don't know. The standard approach is the same as for any hyperparameter: try a range of degrees, evaluate each with cross-validation (never on the training set alone), and pick the degree that minimizes validation error, not training error.

The overfitting cliff at high degrees

This is the central cautionary lesson of polynomial regression: training error always decreases (or stays the same) as you add degree, because a higher-degree polynomial is strictly more flexible and can fit training points more exactly. A degree-(n-1) polynomial can pass through every single point in an n-point dataset with zero training error.

But that perfect fit comes from chasing noise, not signal. Between training points, a high-degree polynomial oscillates wildly — a phenomenon visually obvious the moment you plot it, known as Runge's phenomenon in numerical analysis. The result is a model with near-zero training error and catastrophic error on new data: textbook overfitting, made unusually easy to see because you can literally plot the curve and watch it whipsaw.

Degree vs. behavior

Taming high-degree polynomials with regularization

If you need a higher degree to capture genuine complexity in the data, combine polynomial features with Ridge or Lasso regression instead of plain OLS. The regularization penalty discourages the large, oscillating coefficients that cause high-degree polynomials to misbehave, letting you use a flexible feature set while keeping the fitted curve smooth and well-behaved.

What's next

Polynomial regression shows how far you can push a single flexible model. Boosting Algorithms take a completely different route to flexibility — instead of one complex model, they combine many simple ones sequentially, each correcting the mistakes of the last.