Catastrophic Forgetting

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The problem: learning B can erase A

Neural networks trained sequentially on different tasks exhibit a stubborn failure mode: when the network is trained on task B after already learning task A, performance on task A often collapses — sometimes to near-random — even though nothing about task A changed. This is catastrophic forgetting. It stands in sharp contrast to how humans typically learn: picking up a new skill rarely erases an old one entirely. Understanding why neural networks behave this way, and what can be done about it, is central to continual learning, lifelong learning systems, and any pipeline that updates a deployed model incrementally rather than retraining from scratch each time.

Why it happens: shared, dense parameters

The root cause is that standard neural networks represent what they've learned as distributed patterns across a shared, dense set of weights — there's no dedicated, isolated subset of parameters for 'task A knowledge.' When gradient descent optimizes those same weights for a new task B, it adjusts them in whatever direction reduces B's loss, with no inherent mechanism protecting directions in weight space that happened to matter for A. If A and B pull weights in conflicting directions, A's performance degrades as a direct side effect of B's optimization — not because the network ran out of capacity, but because nothing in vanilla SGD or Adam knows to preserve it.

forgetting_demo.py

python

# Minimal illustration: a shared linear layer trained sequentially
# on two tasks shows forgetting of task A after training on task B.
import torch, torch.nn as nn

model = nn.Linear(10, 2)
opt = torch.optim.SGD(model.parameters(), lr=0.1)

def train_task(X, y, steps=200):
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.functional.cross_entropy(model(X), y)
        loss.backward()
        opt.step()

X_a, y_a = make_task_a_data()
X_b, y_b = make_task_b_data()   # different input/label distribution

train_task(X_a, y_a)
acc_a_before = accuracy(model, X_a, y_a)   # high

train_task(X_b, y_b)             # same weights, no protection for task A
acc_a_after = accuracy(model, X_a, y_a)    # often collapses toward chance

print(f"Task A accuracy: {acc_a_before:.2f} -> {acc_a_after:.2f}")

The stability-plasticity dilemma

Catastrophic forgetting is one face of a deeper, unavoidable tension called the stability-plasticity dilemma: a learning system needs enough plasticity to acquire new information, but enough stability to retain old information, and these pull in opposite directions. A network with maximal plasticity (every weight freely adjustable for the newest task) forgets fast. A network with maximal stability (weights frozen after initial training) cannot learn anything new. Every continual-learning method is, at its core, a particular way of trading off somewhere along this spectrum.

Families of mitigation strategies

Strategy family	Core idea	Example method	Main cost
Regularization-based	Penalize changes to weights important for old tasks	Elastic Weight Consolidation	Requires estimating per-weight importance
Replay-based	Mix old-task data (real or generated) back into new training	Experience replay, generative replay	Needs storage or a generative model for old data
Architectural	Allocate separate capacity per task	Progressive Neural Networks, modular adapters	Model grows with number of tasks
Meta-learning based	Learn an initialization/update rule robust to sequential tasks	Online-aware meta-learning approaches	More complex training procedure

Fine-tuning a deployed LLM is a forgetting risk, not just a capability gain

Any time you fine-tune a model further — for a new domain, a new tone, or a new tool-use format — you're running the exact sequential-learning setup that causes catastrophic forgetting. Teams that fine-tune production LLMs iteratively need to continuously evaluate on the model's original broad capability set, not just the new target task, or they risk silently regressing general performance with every update cycle.

Forgetting isn't always bad

In some contexts, forgetting is desirable — machine unlearning, where a model is deliberately made to forget specific training examples for privacy or legal compliance (e.g., GDPR's 'right to be forgotten'), intentionally exploits similar dynamics. The question of how to forget *precisely* the right information without collateral damage to everything else is itself an active and nontrivial research problem.

What's next?

Elastic Weight Consolidation is one of the most concrete and widely cited answers to this problem — a regularization technique that estimates which weights matter for old tasks and explicitly discourages moving them. It's a useful lens for seeing exactly how the stability-plasticity tradeoff gets implemented in practice.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →