mlsanityData to 7 Oct 2026

Continual Learning with Recursive Gradient Optimization

arXiv 2201.12522 ↗First posted 29 Jan 2022cs.LG cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Learning multiple tasks sequentially without forgetting previous knowledge, called Continual Learning(CL), remains a long-standing challenge for neural networks. Most existing methods rely on additional network capacity or data replay. In contrast, we introduce a novel approach which we refer to as Recursive Gradient Optimization(RGO). RGO is composed of an iteratively updated optimizer that modifies the gradient to minimize forgetting without data replay and a virtual Feature Encoding Layer(FEL) that represents different long-term structures with only task descriptors. Experiments demonstrate that RGO has significantly better performance on popular continual classification benchmarks when compared to the baselines and achieves new state-of-the-art performance on 20-split-CIFAR100(82.22%) and 20-split-miniImageNet(72.63%). With higher average accuracy than Single-Task Learning(STL), this method is flexible and reliable to provide continual learning capabilities for learning models that rely on gradient descent.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 29 Jan 2022.

Builds on

29 of 43 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2017Continual Learning with Deep Generative Replaycited by 1,152
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  10. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  11. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  12. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
17 more
  1. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  2. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  3. 2021Gradient Projection Memory for Continual Learningcited by 338
  4. 2017Variational Continual Learningcited by 335
  5. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  6. 2019Continual learning with hypernetworkscited by 267
  7. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  8. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  9. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  10. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  11. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  12. 2018Modular meta-learningcited by 51
  13. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 47
  14. 2019Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalizationcited by 40
  15. 2018Automatically Composing Representation Transformations as a Means for Generalizationcited by 35
  16. 2020Generative Feature Replay with Orthogonal Weight Modification for Continual Learningcited by 10
  17. 2020Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpointcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 10; 2025: 6; 2026: 5.

32 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  4. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  5. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  6. 2024Make Continual Learning Stronger via C-Flatcited by 35
  7. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  8. 2022Grow and Merge: A Unified Framework for Continuous Categories Discoverycited by 33
  9. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  10. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 26
  11. 2023Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learningcited by 21
  12. 2023Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learningcited by 10
20 more
  1. 2022Progressive Learning without Forgettingcited by 7
  2. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  3. 2023Restricted Orthogonal Gradient Projection for Continual Learningcited by 6
  4. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  5. 2024Maintaining Adversarial Robustness in Continuous Learningcited by 5
  6. 2024CroMo-Mixup: Augmenting Cross-Model Representations for Continual Self-Supervised Learningcited by 4
  7. 2025Rethinking Continual Learning with Progressive Neural Collapsecited by 4
  8. 2025DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspacescited by 3
  9. 2024An Effective Dynamic Gradient Calibration Method for Continual Learningcited by 2
  10. 2024Temporal-Difference Variational Continual Learningcited by 1
  11. 2025Consistency-Driven Calibration and Matching for Few-Shot Class-Incremental Learningcited by 1
  12. 2025C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learningcited by 1
  13. 2025LibContinual: A Comprehensive Library towards Realistic Continual Learningcited by 1
  14. 2026Theoretical Foundations of Continual Learning via Drift-Plus-Penaltycited by 1
  15. 2022Topological Continual Learning with Wasserstein Distance and Barycentercited by 0
  16. 2025SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defensecited by 0
  17. 2026Continual Learning of Achieving Forgetting-free and Positive Knowledge Transfercited by 0
  18. 2026Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repaircited by 0
  19. 2026Understanding Generalization and Forgetting in In-Context Continual Learningcited by 0
  20. 2026Theory of Continual Learning Against Data Poisoning Attackscited by 0