mlsanityData to 14 Oct 2026

Entropy-based Stability-Plasticity for Lifelong Learning

arXiv 2204.09517 ↗First posted 18 Apr 2022cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

The ability to continuously learn remains elusive for deep learning models. Unlike humans, models cannot accumulate knowledge in their weights when learning new tasks, mainly due to an excess of plasticity and the low incentive to reuse weights when training a new task. To address the stability-plasticity dilemma in neural networks, we propose a novel method called Entropy-based Stability-Plasticity (ESP). Our approach can decide dynamically how much each model layer should be modified via a plasticity factor. We incorporate branch layers and an entropy-based criterion into the model to find such factor. Our experiments in the domains of natural language and vision show the effectiveness of our approach in leveraging prior knowledge by reducing interference. Also, in some cases, it is possible to freeze layers during training leading to speed up in training.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

40 of 53 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  9. 2016Progressive Neural Networkscited by 1,361
  10. 2014How transferable are features in deep neural networks?cited by 1,225
  11. 2017Continual Learning with Deep Generative Replaycited by 1,158
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
28 more
  1. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  2. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  3. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  4. 2019Three scenarios for continual learningcited by 614
  5. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  6. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  7. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  8. 2021Gradient Projection Memory for Continual Learningcited by 348
  9. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  10. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  11. 2019Revealing the Dark Secrets of BERTcited by 301
  12. 2019Continual learning with hypernetworkscited by 273
  13. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  14. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  15. 2019Episodic Memory in Lifelong Language Learningcited by 188
  16. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  17. 2020Continual Prototype Evolution: Learning Online from Non-Stationary Data Streamscited by 144
  18. 2020Adversarial Continual Learningcited by 142
  19. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  20. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
  21. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 89
  22. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 72
  23. 2018Generative Models from the perspective of Continual Learningcited by 69
  24. 2021Continual Learning via Local Module Compositioncited by 63
  25. 2020Lifelong Learning of Compositional Structurescited by 35
  26. 2021Optimizing Reusable Knowledge for Continual Learning via Metalearningcited by 25
  27. 2021DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inferencecited by 9
  28. 2020Continual Learning with Deep Artificial Neuronscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 5.

8 papers.

  1. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 16
  2. 2024Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Modelscited by 11
  3. 2023Continual Learning for Predictive Maintenance: Overview and Challengescited by 10
  4. 2024Progress Measures for Grokking on Real-world Taskscited by 8
  5. 2022How Relevant is Selective Memory Population in Lifelong Language Learning?cited by 6
  6. 2023On the Effectiveness of Equivariant Regularization for Robust Online Continual Learningcited by 2
  7. 2024Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exitingcited by 2
  8. 2024Realistic Continual Learning Approach using Pre-trained Modelscited by 0