Time Series Forecasting with Machine Learning

Why time series data breaks the usual rules

Most machine learning assumes your rows are independent and identically distributed — shuffle them, split them randomly, and nothing is lost. Time series data violates that assumption completely. Today's value depends on yesterday's. A random train/test split would let the model peek at the future to predict the past, and your validation metrics would lie to you.

Forecasting is the task of predicting future values of a sequence — sales next month, server load next hour, website traffic next week — using its own history and possibly external signals. It sits at the intersection of statistics and machine learning, and the two traditions approach it differently.

Decomposing a series: trend, seasonality, residual

Before modeling anything, it helps to decompose a series into three components:

Trend — the long-term direction, rising, falling, or flat, often extracted with a moving average or a fitted smooth curve.

Seasonality — a repeating pattern tied to a fixed period (daily, weekly, yearly). Retail sales spike every December; electricity demand peaks every afternoon.

Residual (noise) — whatever is left after removing trend and seasonality: the unpredictable part, ideally close to white noise.

A classical decomposition writes the series additively as $y_t = T_t + S_t + R_t$, or multiplicatively as $y_t = T_t \cdot S_t \cdot R_t$ when the seasonal swings grow with the trend (common in sales data). Visualizing these three components separately is often the single most useful diagnostic step before choosing a model — it tells you whether you even need a sophisticated algorithm or just a trend line plus a seasonal offset.

ARIMA: the classical workhorse

ARIMA (AutoRegressive Integrated Moving Average) is the classical statistical backbone of forecasting. It combines three ideas:

AR(p) — the value today is a linear combination of the previous $p$ values: $y_t = c + \sum_{i=1}^{p} \phi_i y_{t-i} + \epsilon_t$.

I(d) — differencing the series $d$ times to remove trend and make it stationary (constant mean and variance over time), since AR and MA models assume stationarity.

MA(q) — the value today is a linear combination of the previous $q$ forecast errors: $y_t = \mu + \sum_{j=1}^{q} \theta_j \epsilon_{t-j} + \epsilon_t$.

Put together, ARIMA(p, d, q) fits a compact linear model after making the series stationary. Seasonal variants (SARIMA) add a second set of (P, D, Q, s) terms for a seasonal period $s$. ARIMA is interpretable, requires relatively little data, and is still a strong baseline for series that are roughly linear and well-behaved.

When machine learning beats classical forecasting

ARIMA models one series using only its own past. Machine learning models — gradient boosted trees, random forests, and neural networks like LSTMs or temporal convolutional networks — can incorporate many external features at once: day of week, holiday flags, promotions, weather, prices of related products, and even hundreds of other correlated series.

ML tends to win when:
- You have rich exogenous features that classical models can't easily absorb.
- You're forecasting many related series at once (thousands of SKUs) and want a single shared model instead of fitting one ARIMA per series.
- Relationships are nonlinear (e.g., demand response to price is not a straight line).

Classical models tend to win when:
- You have a single series with limited history — tree models and neural nets need more data to learn patterns than a hand-specified AR term does.
- Interpretability matters (a regulator wants to see your model's coefficients, not a forest of decision paths).
- The series is simple and well-behaved; added ML complexity buys nothing.

Classical statistical models vs. machine learning forecasters

Feature engineering for ML-based forecasting

To forecast with a standard regression model (like XGBoost), you reframe the problem as supervised learning: build features from the past to predict a value in the future. Common features include:

- Lag features: the value 1, 7, and 28 days ago.
- Rolling statistics: 7-day rolling mean, rolling standard deviation, rolling min/max.
- Calendar features: day of week, month, is-holiday, days-until-next-holiday.
- Differencing features: change from the previous period, percentage change.

This turns a sequence problem into a tabular one, which is why tree-based models work so well in forecasting competitions — they just need the right lag and rolling features handed to them.

Validating forecasts correctly

Random k-fold cross-validation is invalid for time series — it trains on future data and tests on the past. Instead, use walk-forward validation (also called expanding or rolling window validation): train on data up to time $t$, test on $t+1 \ldots t+h$, then slide the window forward and repeat. This mimics how the model will actually be used in production and gives an honest estimate of forecast error. Common error metrics are MAE, RMSE, and MAPE (mean absolute percentage error), chosen based on whether large errors should be penalized more heavily and whether the series can hit zero (which breaks MAPE).

What's next

Forecasting is a specific flavor of a more general problem: making valid predictions on data with structure that a naive train/test split would violate. The next lesson on cross-validation strategies dives deeper into walk-forward validation and other splitting schemes that keep your evaluation honest across different kinds of structured data.