mlsanityData to 7 Oct 2026

Overcoming catastrophic forgetting with hard attention to the task

arXiv 1801.01423 ↗First posted 4 Jan 2018cs.LG cs.AI cs.NEcited by 672

Not on the map. Why?

Abstract (arXiv)

Catastrophic forgetting occurs when a neural network loses the information learned in a previous task after training on subsequent tasks. This problem remains a hurdle for artificial intelligence systems with sequential learning capabilities. In this paper, we propose a task-based hard attention mechanism that preserves previous tasks' information without affecting the current task's learning. A hard attention mask is learned concurrently to every task, through stochastic gradient descent, and previous masks are exploited to condition such learning. We show that the proposed mechanism is effective for reducing catastrophic forgetting, cutting current rates by 45 to 80%. We also show that it is robust to different hyperparameter choices, and that it offers a number of monitoring capabilities. The approach features the possibility to control both the stability and compactness of the learned knowledge, which we believe makes it also attractive for online learning or network compression applications.

Key facts. Cited by 672 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 84; 2022: 118; 2023: 147; 2024: 141; 2025: 111; 2026: 71.

672 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2021Learning to Prompt for Continual Learningcited by 938
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  5. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  6. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  7. 2021Gradient Projection Memory for Continual Learningcited by 338
  8. 2021Towards Open World Object Detectioncited by 297
  9. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  10. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  11. 2021Co2L: Contrastive Continual Learningcited by 243
  12. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
38 more
  1. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
  2. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  3. 2023Continual Pre-training of Language Modelscited by 194
  4. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  5. 2021Generalized Source-free Domain Adaptationcited by 175
  6. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  7. 2021Self-Supervised Models are Continual Learnerscited by 149
  8. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  9. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  10. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  11. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  12. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  13. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 114
  14. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  15. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  16. 2021Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representationscited by 96
  17. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  18. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  19. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  20. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  21. 2023Machine Unlearning: Solutions and Challengescited by 85
  22. 2021RECALL: Replay-based Continual Learning in Semantic Segmentationcited by 83
  23. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  24. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
  25. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  26. 2022A Theoretical Study on Solving Continual Learningcited by 78
  27. 2021Rehearsal revealed: The limits and merits of revisiting samples in continual learningcited by 76
  28. 2024Continual Learning and Catastrophic Forgettingcited by 73
  29. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 71
  30. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
  31. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  32. 2021Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Datacited by 67
  33. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  34. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  35. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 64
  36. 2021Continual Learning via Local Module Compositioncited by 61
  37. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  38. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 60

50 most cited of 672 All 672 (JSON)