LSTM for Time Series Forecasting
Why plain RNNs struggle with long sequences
A recurrent neural network processes a sequence step by step, maintaining a hidden state $h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$ that's meant to summarize everything seen so far. In theory this lets the network use information from arbitrarily far in the past. In practice, training through many time steps means backpropagating gradients through the same repeated multiplication, and the chain rule compounds: gradients either shrink toward zero (vanishing) or blow up (exploding) as they travel backward, roughly proportional to $(\text{largest eigenvalue of } W_h)^T$ over $T$ steps. Vanishing gradients mean the network effectively can't learn dependencies more than a handful of steps back — a serious problem for time series with weekly or seasonal patterns spanning hundreds of steps.
The LSTM cell: gates that control information flow
Long Short-Term Memory networks (Hochreiter & Schmidhuber, 1997) solve this with a separate cell state $c_t$ that acts as a conveyor belt for information, modified only through addition and element-wise gating rather than repeated matrix multiplication — this is what keeps gradients from vanishing over long spans.
At each step, three sigmoid gates control what happens to the cell state:
Forget gate: $f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$ — decides what fraction of the old cell state to keep.
Input gate: $i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$ — decides how much new information to write in.
Candidate update: $\tilde{c}_t = \tanh(W_c [h_{t-1}, x_t] + b_c)$.
Cell state update: $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$.
Output gate: $o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$, and the hidden state is $h_t = o_t \odot \tanh(c_t)$.
Each gate is a small neural network learned jointly with the rest of the model, so the LSTM learns for itself when to remember, when to forget, and when to expose information to the output.
Framing forecasting as a sequence problem
To use an LSTM for forecasting, you convert the raw series into supervised windows: a lookback window of the last $n$ observations (and optionally exogenous features) as input, and the next one or more future values as the target. The model can be trained to predict one step ahead and then applied recursively (feeding its own prediction back in as input for the next step), or trained directly for multi-step output via a sequence-to-sequence or direct multi-horizon head — the latter avoids the compounding error that comes from recursive one-step forecasting.
Scaling matters a lot here: because gates use sigmoid and tanh, inputs with wildly different magnitudes slow training and hurt stability, so series are typically normalized (min-max or standardization) before windowing.
import torch
import torch.nn as nn
import numpy as np
def make_windows(series, lookback, horizon):
X, y = [], []
for i in range(len(series) - lookback - horizon + 1):
X.append(series[i : i + lookback])
y.append(series[i + lookback : i + lookback + horizon])
return np.array(X), np.array(y)
class LSTMForecaster(nn.Module):
def __init__(self, input_size=1, hidden_size=64, num_layers=2, horizon=7):
super().__init__()
self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
self.head = nn.Linear(hidden_size, horizon)
def forward(self, x):
# x: (batch, lookback, input_size)
out, (h_n, c_n) = self.lstm(x)
last_hidden = out[:, -1, :] # final time step's hidden state
return self.head(last_hidden) # (batch, horizon)
lookback, horizon = 30, 7
series = (np.load("scaled_series.npy")).astype("float32")
X, y = make_windows(series, lookback, horizon)
X_t = torch.tensor(X).unsqueeze(-1) # add feature dim
y_t = torch.tensor(y)
model = LSTMForecaster(horizon=horizon)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()
for epoch in range(50):
optimizer.zero_grad()
pred = model(X_t)
loss = loss_fn(pred, y_t)
loss.backward()
optimizer.step()
GRUs are a lighter alternative with similar gradient behavior
LSTMs need meaningfully more data than classical models
Attention and the limits of pure recurrence
Even with gating, LSTMs still process a sequence strictly step by step, which caps parallelism during training and can still struggle to directly weigh a specific distant past event against the present. Adding an attention mechanism over the LSTM's hidden states lets the model learn to look back at specific relevant time steps directly, rather than relying entirely on the compressed cell state to carry that information forward — a technique borrowed from, and eventually displaced by, Transformer-style architectures.
What's next
LSTMs showed that gating could tame the vanishing gradient problem, but their strictly sequential computation is a real bottleneck for training speed on long sequences. An alternative family of models replaces recurrence with layered convolutions to capture long-range dependencies while processing the whole sequence in parallel — worth understanding as a direct architectural counterpoint to recurrent forecasting.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.