Incremental Learning & Catastrophic Interference

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Learning task 2 shouldn't mean forgetting task 1

Incremental (continual) learning is the setting where a model must learn a sequence of tasks or classes over time, without full access to earlier tasks' data, while retaining performance on everything it learned before. The central obstacle is catastrophic forgetting (McCloskey & Cohen, 1989; rediscovered repeatedly in deep learning): when a neural network is fine-tuned on new data, gradient updates optimized purely for the new task can overwrite the weight configurations that encoded earlier tasks, causing performance on old tasks to collapse — sometimes to near-random — even though nothing about the architecture changed.

Why it happens: shared, distributed representations

Unlike symbolic systems with localized memory, neural networks store knowledge as distributed patterns across shared weights. There's no dedicated 'slot' for task 1's knowledge — the same weight matrix that computed task 1's features gets overwritten by gradients computed entirely from task 2's loss, $\nabla_\theta \mathcal{L}_2(\theta)$, with no term in the objective that penalizes moving away from the task-1-optimal region. The more tasks share architecture (as opposed to having separate dedicated parameters), the more interference occurs — this is the classic stability-plasticity dilemma: a model needs enough plasticity to learn new things and enough stability to retain old things, and standard SGD gives you all plasticity, zero stability.

Approach family	Mechanism	Example method
Regularization-based	Penalize changes to weights important for old tasks	EWC (Elastic Weight Consolidation), SI
Replay-based	Interleave old examples (real or generated) with new ones	Experience replay, generative replay, iCaRL
Architecture-based	Allocate new capacity per task, freeze old parameters	Progressive Neural Networks, PackNet
Parameter-isolation	Separate subnetworks/masks per task within one model	PackNet, HAT (Hard Attention to the Task)

Elastic Weight Consolidation: the regularization approach

EWC (Kirkpatrick et al., 2017) estimates, via the diagonal of the Fisher information matrix, how important each weight was for previous tasks, then adds a quadratic penalty to the new task's loss that discourages moving important weights far from their old values: $\mathcal{L}(\theta) = \mathcal{L}_{new}(\theta) + \frac{\lambda}{2}\sum_i F_i (\theta_i - \theta_{i,old}^*)^2$. Weights with high Fisher information (i.e., the loss is very sensitive to them) get pulled back strongly if they drift; weights that barely mattered for the old task are left free to adapt. This is elegant but has limits: it only protects against forgetting the immediately preceding task well unless importance is accumulated across the whole sequence, and the quadratic approximation can be inaccurate for highly nonlinear loss landscapes.

ewc_penalty.py

python

import torch

def fisher_diagonal(model, dataloader, loss_fn):
    """Estimate per-parameter importance via squared gradients (empirical Fisher)."""
    fisher = {n: torch.zeros_like(p) for n, p in model.named_parameters()}
    model.eval()
    for x, y in dataloader:
        model.zero_grad()
        loss = loss_fn(model(x), y)
        loss.backward()
        for n, p in model.named_parameters():
            fisher[n] += p.grad.detach() ** 2 / len(dataloader)
    return fisher

def ewc_loss(model, old_params, fisher, lam=1000.0):
    """Quadratic penalty pulling important weights back toward old-task values."""
    penalty = 0.0
    for n, p in model.named_parameters():
        penalty += (fisher[n] * (p - old_params[n]) ** 2).sum()
    return lam * penalty

# total_loss = new_task_loss + ewc_loss(model, old_params, fisher)

Replay: the simplest thing that actually works well

Replay-based methods sidestep the math entirely by keeping (or generating) a small buffer of old-task examples and mixing them back into every new-task training batch — effectively turning the sequential problem back into something closer to joint training. iCaRL combines a bounded exemplar buffer with a nearest-class-mean classifier and knowledge distillation from the old model's outputs. Generative replay avoids storing raw data (useful under privacy constraints) by training a generative model to reproduce old-task data distributions on demand. Empirically, replay methods tend to outperform pure regularization methods when even a small buffer is affordable, but they don't scale cleanly to settings where storing or generating old data is infeasible (e.g., strict data-deletion requirements).

Forgetting isn't always bad

In genuinely non-stationary environments, 'forgetting' stale patterns can be a feature, not a bug — this is exactly the tension between continual learning (which wants to retain everything) and online learning / drift adaptation (which wants to let go of outdated patterns). The right amount of plasticity vs. stability is task-dependent, not a universal constant.

Benchmarks can overstate real-world difficulty

Many continual learning papers benchmark on artificially partitioned datasets (e.g., Split-MNIST, Split-CIFAR) with sharp task boundaries and known task identity at test time. Real deployments rarely have such clean boundaries — task-agnostic, boundary-free continual learning is substantially harder and much less mature.

What's Next?

Continual learning on a single machine is one problem; keeping models fresh and consistent when the data itself is arriving continuously, at scale, through a live production pipeline introduces its own engineering challenges.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →