mlsanityData to 7 Oct 2026

Continual Learning for Recurrent Neural Networks: an Empirical Evaluation

arXiv 2103.07492 ↗First posted 12 Mar 2021cs.LG cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

Learning continuously during all model lifetime is fundamental to deploy machine learning solutions robust to drifts in the data distribution. Advances in Continual Learning (CL) with recurrent neural networks could pave the way to a large number of applications where incoming data is non stationary, like natural language processing and robotics. However, the existing body of work on the topic is still fragmented, with approaches which are application-specific and whose assessment is based on heterogeneous learning protocols and datasets. In this paper, we organize the literature on CL for sequential data processing by providing a categorization of the contributions and a review of the benchmarks. We propose two new benchmarks for CL with sequential data based on existing datasets, whose characteristics resemble real-world applications. We also provide a broad empirical evaluation of CL and Recurrent Neural Networks in class-incremental scenario, by testing their ability to mitigate forgetting with a number of different strategies which are not specific to sequential data processing. Our results highlight the key role played by the sequence length and the importance of a clear specification of the CL scenario.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 71 of its 127 references that are in the map. First posted on arXiv 12 Mar 2021.

Builds on

71 of 127 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2016Learning without Forgettingcited by 2,465
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  10. 2016Progressive Neural Networkscited by 1,332
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  12. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
38 more
  1. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  5. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  7. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  8. 2014Neural Turing Machinescited by 630
  9. 2019Three scenarios for continual learningcited by 610
  10. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  11. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  12. 2017Variational Continual Learningcited by 335
  13. 2017A Neural Representation of Sketch Drawingscited by 311
  14. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  15. 2019Continual learning with hypernetworkscited by 267
  16. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
  17. 2018Meta-Learning: A Surveycited by 254
  18. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  19. 2018Task-Free Continual Learningcited by 219
  20. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  21. 2019Meta-Learning Representations for Continual Learningcited by 189
  22. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  23. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  24. 2018Towards Robust Evaluations of Continual Learningcited by 148
  25. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
  26. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  27. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  28. 2018Continuous Learning in Single-Incremental-Task Scenarioscited by 109
  29. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 103
  30. 2019Continual Learning via Neural Pruningcited by 101
  31. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 100
  32. 2020Learning to Continually Learncited by 82
  33. 2018Task Agnostic Continual Learning Using Online Variational Bayescited by 61
  34. 2019Task Agnostic Continual Learning via Meta Learningcited by 53
  35. 2020SpaceNet: Make Free Space For Continual Learningcited by 53
  36. 2019Continuous Meta-Learning without Taskscited by 50
  37. 2018Selfless Sequential Learningcited by 44
  38. 2018Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organizationcited by 38

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 7; 2023: 7; 2024: 5; 2025: 6; 2026: 2.

34 papers.

Sort:

By area:

  1. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  2. 2022Continual evaluation for lifelong learning: Identifying the stability gapcited by 55
  3. 2021Is Class-Incremental Enough for Continual Learning?cited by 22
  4. 2024Continual Learning on Graphs: A Surveycited by 11
  5. 2022Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challengescited by 10
  6. 2023Continual Learning for Predictive Maintenance: Overview and Challengescited by 10
  7. 2021Continual Learning with Echo State Networkscited by 8
  8. 2023Lifelong Continual Learning for Anomaly Detection: New Challenges, Perspectives, and Insightscited by 8
  9. 2023DOCTOR: A Multi-Disease Detection Continual Learning Framework Based on Wearable Medical Sensorscited by 7
  10. 2022Continual Learning of Dynamical Systems with Competitive Federated Reservoir Computingcited by 6
  11. 2021Continual learning of longitudinal health recordscited by 5
  12. 2025Toward Lifelong Learning in Equilibrium Propagation: Sleep-like and Awake Rehearsal for Enhanced Stabilitycited by 5
22 more
  1. 2021TEACHING -- Trustworthy autonomous cyber-physical applications through human-centred intelligencecited by 4
  2. 2021Sustainable Artificial Intelligence through Continual Learningcited by 4
  3. 2024Continual Policy Distillation of Reinforcement Learning-based Controllers for Soft Robotic In-Hand Manipulationcited by 4
  4. 2025VLM-C4L: Continual Core Dataset Learning with Corner Case Optimization via Vision-Language Models for Autonomous Drivingcited by 4
  5. 2021Reminding the Incremental Language Model via Data-Free Self-Distillationcited by 3
  6. 2022Continual Learning for Human State Monitoringcited by 2
  7. 2023A Protocol for Continual Explanation of SHAPcited by 2
  8. 2023A Natural Gas Consumption Forecasting System for Continual Learning Scenarios based on Hoeffding Trees with Change Point Detection Mechanismcited by 2
  9. 2022Continual Learning of Multi-modal Dynamics with External Memorycited by 1
  10. 2022Biologically-Inspired Continual Learning of Human Motion Sequencescited by 1
  11. 2023Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognitioncited by 1
  12. 2023Creating Synthetic Datasets for Collaborative Filtering Recommender Systems using Generative Adversarial Networkscited by 1
  13. 2024Learning to Learn without Forgetting using Attentioncited by 1
  14. 2025Minion Gated Recurrent Unit for Continual Learningcited by 1
  15. 2025Separating the what and how of compositional computation to enable reuse and continual learningcited by 1
  16. 2022Towards lifelong learning of Recurrent Neural Networks for control designcited by 0
  17. 2024PAGE: Domain-Incremental Adaptation with Past-Agnostic Generative Replay for Smart Healthcarecited by 0
  18. 2024ODEStream: A Buffer-Free Online Learning Framework with ODE-based Adaptor for Streaming Time Series Forecastingcited by 0
  19. 2025Learning Dynamic Representations via An Optimally-Weighted Maximum Mean Discrepancy Optimization Framework for Continual Learningcited by 0
  20. 2025Teleology-Driven Affective Computing: A Causal Framework for Sustained Well-Beingcited by 0
  21. 2026Frequency-Domain Neural ODEs for Modeling Non-Linear Dynamical Systemscited by 0
  22. 2026Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gatingcited by 0