mlsanityData to 14 Oct 2026

How Efficient Are Today's Continual Learning Algorithms?

arXiv 2303.18171 ↗First posted 29 Mar 2023cs.CV cs.AI cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Supervised Continual learning involves updating a deep neural network (DNN) from an ever-growing stream of labeled data. While most work has focused on overcoming catastrophic forgetting, one of the major motivations behind continual learning is being able to efficiently update a network with new information, rather than retraining from scratch on the training dataset as it grows over time. Despite recent continual learning methods largely solving the catastrophic forgetting problem, there has been little attention paid to the efficiency of these algorithms. Here, we study recent methods for incremental class learning and illustrate that many are highly inefficient in terms of compute, memory, and storage. Some methods even require more compute than training from scratch! We argue that for continual learning to have real-world applicability, the research community cannot ignore the resources used by these algorithms. There is more to continual learning than mitigating catastrophic forgetting.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

25 of 30 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  6. 2019Large Scale Incremental Learningcited by 732
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  8. 2018End-to-End Incremental Learningcited by 597
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  10. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  11. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  12. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
13 more
  1. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  2. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  3. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  4. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  5. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  6. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  7. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
  8. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
  9. 2019Continual Learning via Neural Pruningcited by 101
  10. 2022Online Continual Learning for Embedded Devicescited by 49
  11. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 48
  12. 2021CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparabilitycited by 28
  13. 2021Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractorscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 9; 2025: 5; 2026: 4.

24 papers.

Sort:

  1. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  2. 2023Continual Learning: Applications and the Road Forwardcited by 59
  3. 2023Online Continual Learning Without the Storage Constraintcited by 29
  4. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  5. 2023GRASP: A Rehearsal Policy for Efficient Online Continual Learningcited by 14
  6. 2024One-stage Prompt-based Continual Learningcited by 14
  7. 2023Overcoming the Stability Gap in Continual Learningcited by 13
  8. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  9. 2025Forget Forgetting: Continual Learning in a World of Abundant Memorycited by 11
  10. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
  11. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 8
  12. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
12 more
  1. 2023Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluationcited by 5
  2. 2024How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation modelscited by 5
  3. 2023Hadamard Domain Training with Integers for Class Incremental Quantized Learningcited by 3
  4. 2025CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptationcited by 2
  5. 2026Tree of Concepts: Interpretable Continual Learners in Non-Stationary Clinical Domainscited by 2
  6. 2024CRoP: Context-wise Robust Static Human-Sensing Personalizationcited by 1
  7. 2025A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initializationcited by 1
  8. 2025A Study on Regularization-Based Continual Learning Methods for Indic ASRcited by 1
  9. 2026Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learningcited by 1
  10. 2026Continual Learning via Self-Probe Gradientscited by 1
  11. 2024Recommendation of data-free class-incremental learning algorithms by simulating future datacited by 0
  12. 2026HydraCIL: Decoupled Class-Incremental Learning through Prototype-Guided Multi-Head Classifierscited by 0