What term describes the tendency of a neural network to abruptly lose previously learned capabilities when it is trained on a new task or data distribution?
Show answer
Catastrophic forgetting
Catastrophic forgetting is the central obstacle in continual learning: gradient updates for new data overwrite weights encoding old knowledge. Mitigations include replaying a fraction of old data (experience replay), regularizing important weights, and the empirical fact that larger, overparameterized models forget less.
Also asked as:
- What term refers to the phenomenon where gradient updates for a new task overwrite the weights encoding previously learned knowledge?