Order parameters and phase transitions of continual learning in deep neural networks

• 成果类型: Article
• 署名作者: Shan, Haozhe; Li, Qianyi; Sompolinsky, Haim
• 署名单位: Harvard University; Harvard University; Harvard Medical School; Columbia University; Columbia University; Harvard University; Hebrew University of Jerusalem
• 刊物名称: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
• ISSN/ISSBN: 0027-8424; 1091-6490
• DOI: 10.1073/pnas.2501899123
• 发表日期: 2026-02-10
• 页码: e2501899123
• 关键词: continual learning; Deep neural network; catastrophic forgetting; anterograde interference; order parameters

摘要

Continual learning (CL) enables animals to learn new tasks without erasing prior knowledge. CL in artificial neural networks (NNs) is challenging due to catastrophic forgetting, where new learning degrades performance on older tasks. While various techniques exist to mitigate forgetting, theoretical insights into when and why CL fails in NNs are lacking. Here, we present a statistical-mechanics theory of CL in deep, wide NNs, which characterizes the network's input-output mapping as it learns a sequence of tasks. It gives rise to order parameters (OPs) that capture how task relations and network architecture influence forgetting and anterograde interference, as verified by numerical evaluations. For networks with a shared readout for all tasks (single-head CL), the relevant-feature and rule similarity between tasks, respectively measured by two OPs, are sufficient to predict a wide range of CL behaviors on classic benchmark tasks. In addition, the theory predicts that increasing the network depth can effectively reduce interference between tasks, thereby lowering forgetting. For networks with task-specific readouts (multihead CL), the theory identifies a phase transition where CL performance shifts dramatically as tasks become less similar, as measured by another task-similarity OP. While forgetting is relatively mild compared to single-head CL across all tasks, sufficiently low similarity leads to catastrophic anterograde interference, where the network retains old tasks and interpolates new training data perfectly but completely fails to generalize new learning. Our results delineate important factors affecting CL performance and offer theoretical insights into common heuristics for mitigation of forgetting.