mlsanityData to 7 Oct 2026

Efficient Lifelong Learning with A-GEM

arXiv 1812.00420 ↗First posted 2 Dec 2018cs.LG stat.MLcited by 1,044

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In lifelong learning, the learner is presented with a sequence of tasks, incrementally building a data-driven prior which may be leveraged to speed up learning of a new task. In this work, we investigate the efficiency of current lifelong approaches, in terms of sample complexity, computational and memory cost. Towards this end, we first introduce a new and a more realistic evaluation protocol, whereby learners observe each example only once and hyper-parameter selection is done on a small and disjoint set of tasks, which is not used for the actual learning experience and evaluation. Second, we introduce a new metric measuring how quickly a learner acquires a new skill. Third, we propose an improved version of GEM (Lopez-Paz & Ranzato, 2017), dubbed Averaged GEM (A-GEM), which enjoys the same or even better performance as GEM, while being almost as computationally and memory efficient as EWC (Kirkpatrick et al., 2016) and other regularization-based methods. Finally, we show that all algorithms including A-GEM can learn even more quickly if they are provided with task descriptors specifying the classification tasks under consideration. Our experiments on several standard lifelong learning benchmarks demonstrate that A-GEM has the best trade-off between accuracy and efficiency.

Key facts. Cited by 1,044 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 148; 2022: 178; 2023: 211; 2024: 198; 2025: 159; 2026: 150.

1,044 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  2. 2021Learning to Prompt for Continual Learningcited by 938
  3. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  4. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  5. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  6. 2021Gradient Projection Memory for Continual Learningcited by 338
  7. 2021Towards Open World Object Detectioncited by 297
  8. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  9. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  10. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  11. 2021Co2L: Contrastive Continual Learningcited by 243
  12. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
38 more
  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  3. 2023Class-Incremental Learning: A Surveycited by 209
  4. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  5. 2022Patching open-vocabulary models by interpolating weightscited by 187
  6. 2021Few-Shot Incremental Learning with Continually Evolved Classifierscited by 184
  7. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  8. 2023Mitigating the Alignment Tax of RLHFcited by 181
  9. 2021Self-Supervised Models are Continual Learnerscited by 149
  10. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  11. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  12. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  13. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  14. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 130
  15. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  16. 2022Continual Learning and Private Unlearningcited by 124
  17. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  18. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  19. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  20. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 114
  21. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  22. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  23. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
  24. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  25. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  26. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  27. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  28. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  29. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 92
  30. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  31. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  32. 2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 87
  33. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  34. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  35. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  36. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  37. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
  38. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79

50 most cited of 1,044 All 1,044 (JSON)