Researchers proposed Local Support Learning to prevent catastrophic forgetting in large models. It pairs a weight adapter with a GMM-based gate. The gate enables the adapter only on inputs from its training distribution. This keeps updates local to the current data. The method retains prior capabilities across multiple training phases. It works on models up to 7 billion parameters. Retention improves with model scale. It decouples learning hyperparameters from forgetting. The approach adds minimal memory and latency overhead. It is robust to hyperparameter choices.
What changed
Gradient-based updates affected all inputs, causing catastrophic forgetting across learning phases.
What it unlocks
Retaining prior model capabilities while learning new tasks without access to prior data.
- 97% retention at 3B parameters
- 99% retention at 7B parameters
- 6.9 MB memory per phase
- 21% inference latency increase
What you need to act on it
- a pre-trained large language model
- a GMM-based gate trained on current phase data
Sources