mlsanityData to 7 Oct 2026

Rainbow Memory: Continual Learning with a Memory of Diverse Samples

arXiv 2103.17230 ↗First posted 31 Mar 2021cs.CV cs.LGcited by 273

Not on the map. Why?

Abstract (arXiv)

Continual learning is a realistic learning scenario for AI models. Prevalent scenario of continual learning, however, assumes disjoint sets of classes as tasks and is less realistic rather artificial. Instead, we focus on 'blurry' task boundary; where tasks shares classes and is more realistic and practical. To address such task, we argue the importance of diversity of samples in an episodic memory. To enhance the sample diversity in the memory, we propose a novel memory management strategy based on per-sample classification uncertainty and data augmentation, named Rainbow Memory (RM). With extensive empirical validations on MNIST, CIFAR10, CIFAR100, and ImageNet datasets, we show that the proposed method significantly improves the accuracy in blurry continual learning setups, outperforming state of the arts by large margins despite its simplicity. Code and data splits will be available in https://github.com/clovaai/rainbow-memory.

Key facts. Cited by 273 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

35 of 48 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2016Learning without Forgettingcited by 2,465
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  12. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
23 more
  1. 2017Continual Learning with Deep Generative Replaycited by 1,152
  2. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  3. 2019Large Scale Incremental Learningcited by 727
  4. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  5. 2018End-to-End Incremental Learningcited by 596
  6. 2019Gradient based sample selection for online continual learningcited by 568
  7. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  8. 2020Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixupcited by 260
  9. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  10. 2019Fast AutoAugmentcited by 243
  11. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  12. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  13. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  14. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 160
  15. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  16. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
  17. 2020GAN Memory with No Forgettingcited by 84
  18. 2018Memory Replay GANs: learning to generate images from new categories without forgettingcited by 76
  19. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  20. 2017Continual Learning in Generative Adversarial Netscited by 51
  21. 2020Online Continual Learning under Extreme Memory Constraintscited by 48
  22. 2020On the Generalization Effects of Linear Transformations in Data Augmentationcited by 47
  23. 2020The Effectiveness of Memory Replay in Large Scale Continual Learningcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 35; 2023: 69; 2024: 62; 2025: 58; 2026: 35.

273 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  3. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  4. 2023Class-Incremental Learning: A Surveycited by 209
  5. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  6. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  7. 2022Constrained Few-shot Class-incremental Learningcited by 93
  8. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  9. 2022A Theoretical Study on Solving Continual Learningcited by 78
  10. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  11. 2022Representation Compensation Networks for Continual Semantic Segmentationcited by 72
  12. 2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 66
38 more
  1. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  2. 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
  3. 2022Class-Incremental Learning with Strong Pre-trained Modelscited by 60
  4. 2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
  5. 2023Continual Detection Transformer for Incremental Object Detectioncited by 59
  6. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  7. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  8. 2022Information-theoretic Online Memory Selection for Continual Learningcited by 42
  9. 2024SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detectioncited by 42
  10. 2023When Prompt-based Incremental Learning Does Not Meet Strong Pretrainingcited by 41
  11. 2023A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Taskscited by 41
  12. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  13. 2023Continual Semantic Segmentation with Automatic Memory Sample Selectioncited by 39
  14. 2023EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understandingcited by 39
  15. 2021Continually Learning Self-Supervised Representations with Projected Functional Regularizationcited by 38
  16. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  17. 2023Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuningcited by 35
  18. 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
  19. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  20. 2022A simple but strong baseline for online continual learning: Repeated Augmented Rehearsalcited by 32
  21. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  22. 2022Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundariescited by 31
  23. 2023Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matchingcited by 29
  24. 2023Online Continual Learning Without the Storage Constraintcited by 29
  25. 2024BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptationcited by 25
  26. 2023ICICLE: Interpretable Class Incremental Continual Learningcited by 24
  27. 2023Dealing with Cross-Task Class Discrimination in Online Continual Learningcited by 24
  28. 2023DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Cluescited by 23
  29. 2024VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Modelcited by 23
  30. 2022Task-Free Continual Learning via Online Discrepancy Distance Learningcited by 22
  31. 2023Open-World Continual Learning: Unifying Novelty Detection and Continual Learningcited by 22
  32. 2022Online Continual Learning with Contrastive Vision Transformercited by 21
  33. 2023Regularizing Second-Order Influences for Continual Learningcited by 21
  34. 2023A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Applicationcited by 21
  35. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  36. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Modelscited by 21
  37. 2022Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learnerscited by 20
  38. 2022Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Datacited by 19

50 most cited of 273 All 273 (JSON)