Continual and Lifelong Learning

Continual learning (also called lifelong learning or incremental learning) is the study of AI systems that learn from a non-stationary stream of tasks or data over time, updating their knowledge without losing previously acquired skills. It is one of the key gaps between current AI and human cognition — humans can learn new things all day without forgetting their native language.

The Core Challenge: Catastrophic Forgetting

When a neural network is trained on task B after having learned task A, weights that were important for task A get overwritten by gradients flowing from task B. Performance on task A can drop to near-random within a single epoch of training on new data. This is catastrophic forgetting (also called catastrophic interference).

It occurs because standard gradient descent minimizes loss on the current data distribution with no mechanism for preserving prior knowledge.

Why This Is Hard

• Shared weights: A single feedforward model stores all knowledge in the same weight tensors — new learning necessarily perturbs old representations
• No explicit task boundary signal: In real-world streams, the model often doesn’t know when the task changes
• Stability-plasticity tradeoff: A model must be plastic enough to learn new things and stable enough to retain old ones — a fundamental tension

Three Families of Solutions

1. Regularization-Based Methods

Identify which weights are important for previous tasks and penalize change to those weights.

Elastic Weight Consolidation (EWC)

EWC (Kirkpatrick et al., 2017) adds a regularization term to the loss:

$$\mathcal{L}_\text{EWC} = \mathcal{L}_B(\theta) + \frac{\lambda}{2} \sum_i F_i (\theta_i - \theta_A^*{}_i)^2$$

where $F_i$ is the Fisher information of parameter $i$ — an estimate of how important parameter $i$ was for task A. Parameters critical to task A are penalized heavily for deviating from their task A values $\theta_A^*$.

Synaptic Intelligence (SI)

SI computes importance online during training — tracking the cumulative contribution of each parameter to the loss reduction, without requiring a separate Fisher computation pass after each task.

Progressive Bayesian Updates

Bayesian approaches maintain a posterior distribution over weights after each task, using it as the prior for the next task. This naturally encodes uncertainty and resistance to change in well-trained parameters.

2. Replay and Memory-Based Methods

Maintain a memory buffer of past examples and interleave them with new training data.

Experience Replay (ER)

Store a subset of old examples in a fixed-size buffer and add them to each training minibatch:

Minibatch = new_samples ∪ sample(memory_buffer)

Simple but effective. The challenge is the buffer is always incomplete — only a fraction of past data can be stored.

Generative Replay

Instead of storing raw data, train a generative model to replay synthetic past examples. The generative model itself must be updated continually — a recursive problem, but it sidesteps memory/privacy constraints.

Dark Experience Replay (DER++)

Store not just examples but also the model’s logit outputs (knowledge distillation targets) at the time of training. Replay both the examples and the original output distribution:

$$\mathcal{L} = \mathcal{L}_\text{ce}(y, f(x)) + \alpha |\hat{z} - f(x)|^2$$

This preserves the model’s prediction behavior on old tasks more precisely than raw label replay.

3. Architecture-Based Methods

Explicitly partition or expand model capacity across tasks.

Progressive Neural Networks

A new “column” of the network is instantiated for each new task, with lateral connections to all previous columns. Previous columns are frozen — they cannot forget because they cannot change.

Limitation: Model grows indefinitely; doesn’t scale to many tasks.

PackNet

After learning each task, prune a subset of weights that are free to be retrained on the next task. Pack multiple tasks into the same network by using disjoint weight subsets.

Prompt-Based Continual Learning (L2P, DualPrompt)

Leverages frozen pre-trained Vision Transformers / LLMs as a stable backbone. Only small task-specific prompt vectors are learned for each task, leaving the main model parameters unchanged. Catastrophic forgetting is avoided because the base model is never updated.

This approach benefits from large pre-trained models as general-purpose feature extractors.

Continual Learning Scenarios

Class-incremental learning is the most challenging — the model must differentiate between all old and new classes at test time without task identifiers.

Metrics for Evaluation

Continual Learning in LLMs

Pre-trained LLMs face continual learning challenges when:

• Knowledge cutoff updates: Training on newer data without forgetting older knowledge
• Instruction tuning drift: Fine-tuning for new tasks degrading base model capabilities
• Personalization: Adapting to individual users over time

Approaches in the LLM context:

• Continual pre-training with replay of old data proportionally mixed
• LoRA-based continual fine-tuning: Learning task-specific LoRA adapters that don’t interfere with shared parameters
• Parameter isolation per domain: Routing inputs to specialized adapter modules

Comparison to Human Learning

Human brains address the stability-plasticity tradeoff through:

• Hippocampal replay: Consolidating experiences during sleep into neocortical long-term storage
• Neuromodulation: Changing learning rates contextually via acetylcholine and dopamine
• Sparse coding: Activating few neurons per stimulus, reducing overlap between different memories

Complementary Learning Systems (CLS) theory — a key neuroscience model — has directly inspired replay-based continual learning algorithms.

Key Benchmarks

• Split CIFAR-10/100: CIFAR images partitioned into sequential tasks
• Permuted MNIST: MNIST with pixel permutations as tasks
• CORe50: Continuous learning from video streams of real objects
• CLEAR Benchmark: Real-world, temporally ordered image streams

Further Reading

• Kirkpatrick et al. (2017), Overcoming Catastrophic Forgetting in Neural Networks (EWC)
• Zenke et al. (2017), Continual Learning Through Synaptic Intelligence
• Lopez-Paz & Ranzato (2017), Gradient Episodic Memory for Continual Learning
• Wang et al. (2022), Learning to Prompt for Continual Learning (L2P)
• De Lange et al. (2022), A Continual Learning Survey: Defying Forgetting in Classification Tasks