Elastic Weight Consolidation
Protecting important weights, not all weights
Elastic Weight Consolidation (EWC), introduced by DeepMind researchers in 2017, is a regularization-based approach to catastrophic forgetting. The intuition: not every weight in a network matters equally for a given task. Some weights, if changed even slightly, badly hurt performance on task A; others can shift substantially with little effect. If you can identify which weights matter most for A, you can let the network adjust the unimportant ones freely while training on task B, and penalize changes to the important ones — allowing new learning without destroying what mattered before.
Measuring importance with the Fisher information matrix
EWC estimates each weight's importance using the diagonal of the Fisher information matrix, computed from task A's training data after training on A completes. Intuitively, the Fisher information for weight $\theta_i$ measures how sensitive the model's output distribution is to small changes in $\theta_i$ — a high value means even a small perturbation substantially changes predictions, so that weight was doing important work for task A. Fisher information is estimated as the expected squared gradient of the log-likelihood with respect to each parameter, averaged over task A's data, and is cheap to compute because it only requires gradients already available from standard backpropagation.
The EWC loss
When training on a new task B, EWC adds a quadratic penalty term to the normal task loss:
$\mathcal{L}(\theta) = \mathcal{L}_B(\theta) + \frac{\lambda}{2} \sum_i F_i (\theta_i - \theta_{A,i}^*)^2$
where $\mathcal{L}_B$ is the ordinary loss on the new task, $\theta_{A,i}^*$ is the value weight $i$ had right after training on task A, $F_i$ is that weight's Fisher information (importance), and $\lambda$ is a hyperparameter controlling how strongly old-task importance is enforced. The effect: weights with high Fisher information are pulled back toward their task-A values whenever task B's training pushes them away, while low-importance weights are left essentially unconstrained.
import torch
def compute_fisher(model, dataloader, loss_fn):
"""Estimate diagonal Fisher information after training on task A."""
fisher = {n: torch.zeros_like(p) for n, p in model.named_parameters()}
model.eval()
for x, y in dataloader:
model.zero_grad()
loss = loss_fn(model(x), y)
loss.backward()
for n, p in model.named_parameters():
if p.grad is not None:
fisher[n] += p.grad.detach() ** 2 / len(dataloader)
return fisher
def ewc_penalty(model, fisher, old_params, lam=1000.0):
penalty = 0.0
for n, p in model.named_parameters():
penalty += (fisher[n] * (p - old_params[n]) ** 2).sum()
return 0.5 * lam * penalty
# Training on task B:
# total_loss = task_b_loss + ewc_penalty(model, fisher_a, params_after_task_a)EWC is quadratic regularization anchored to an old solution
EWC doesn't scale perfectly to many sequential tasks
EWC vs. simpler baselines
| Method | Mechanism | Forgetting resistance | Extra storage |
|---|---|---|---|
| Naive fine-tuning | No protection at all | Poor | None |
| L2 regularization to old weights | Penalizes all weight changes equally | Mediocre — doesn't distinguish important weights | Old weight snapshot |
| Elastic Weight Consolidation | Penalizes changes weighted by Fisher importance | Good for a small number of tasks | Old weights + Fisher diagonal per task |
What's next?
EWC represents the regularization branch of continual-learning solutions; replay-based and architectural approaches tackle the same stability-plasticity tradeoff from different angles, and combining ideas across these families is where a lot of practical continual-learning systems end up — there's rarely a single mechanism that solves the problem outright.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.