mlsanityData to 14 Oct 2026

Meta-Learning Representations for Continual Learning

arXiv 1905.12588 ↗First posted 29 May 2019cs.LG cs.AI stat.MLcited by 189

Not on the map. Why?

Abstract (arXiv)

A continual learning agent should be able to build on top of existing knowledge to learn on new data quickly while minimizing forgetting. Current intelligent systems based on neural network function approximators arguably do the opposite---they are highly prone to forgetting and rarely trained to facilitate future learning. One reason for this poor behavior is that they learn from a representation that is not explicitly trained for these two goals. In this paper, we propose OML, an objective that directly minimizes catastrophic interference by learning representations that accelerate future learning and are robust to forgetting under online updates in continual learning. We show that it is possible to learn naturally sparse representations that are more effective for online updating. Moreover, our algorithm is complementary to existing continual learning strategies, such as MER and GEM. Finally, we demonstrate that a basic online updating strategy on representations learned by OML is competitive with rehearsal based methods for continual learning. We release an implementation of our method at https://github.com/khurramjaved96/mrcl .

Key facts. Cited by 189 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 57; 2022: 32; 2023: 35; 2024: 24; 2025: 21; 2026: 20.

189 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  3. 2021Co2L: Contrastive Continual Learningcited by 244
  4. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  5. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  6. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  7. 2021Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representationscited by 96
  8. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  9. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 81
  10. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  11. 2021ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learningcited by 68
  12. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
38 more
  1. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  2. 2021Continual Learning via Local Module Compositioncited by 63
  3. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  4. 2021Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworkscited by 56
  5. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  6. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  7. 2022Memory Efficient Continual Learning with Transformerscited by 43
  8. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  9. 2023MetaGCD: Learning to Continually Learn in Generalized Category Discoverycited by 42
  10. 2021DualNet: Continual Learning, Fast and Slowcited by 39
  11. 2021Understanding Continual Learning Settings with Data Distribution Drift Analysiscited by 38
  12. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 38
  13. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
  14. 2021Wanderlust: Online Continual Object Detection in the Real Worldcited by 34
  15. 2021Learning where to learn: Gradient sparsity in meta and continual learningcited by 33
  16. 2022The Effect of Task Ordering in Continual Learningcited by 31
  17. 2021Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffercited by 30
  18. 2023Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulationcited by 30
  19. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillationcited by 29
  20. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  21. 2021Learngene: From Open-World to Your Learning Taskcited by 27
  22. 2021Continual Learning on Noisy Data Streams via Self-Purified Replaycited by 27
  23. 2021Meta-Learning to Improve Pre-Trainingcited by 27
  24. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  25. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  26. 2021Optimizing Reusable Knowledge for Continual Learning via Metalearningcited by 25
  27. 2021Block Contextual MDPs for Continual Learningcited by 25
  28. 2022Continual Object Detection: A review of definitions, strategies, and challengescited by 25
  29. 2021Formalizing the Generalization-Forgetting Trade-off in Continual Learningcited by 24
  30. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  31. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  32. 2024Continual Learning on Graphs: Challenges, Solutions, and Opportunitiescited by 21
  33. 2021Prototype-Guided Memory Replay for Continual Learningcited by 19
  34. 2021Convolutional Neural Networks Are Not Invariant to Translation, but They Can Learn to Becited by 19
  35. 2022On-device Training: A First Overview on Existing Systemscited by 19
  36. 2023Prediction and Control in Continual Reinforcement Learningcited by 19
  37. 2021Continual Learning for Image-Based Camera Localizationcited by 18
  38. 2023Is forgetting less a good inductive bias for forward transfer?cited by 18

50 most cited of 189 All 189 (JSON)