mlsanityData to 7 Oct 2026

Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation

arXiv 2209.06107 ↗First posted 12 Sep 2022cs.LG cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Continual learning aims to rapidly and continually learn the current task from a sequence of tasks. Compared to other kinds of methods, the methods based on experience replay have shown great advantages to overcome catastrophic forgetting. One common limitation of this method is the data imbalance between the previous and current tasks, which would further aggravate forgetting. Moreover, how to effectively address the stability-plasticity dilemma in this setting is also an urgent problem to be solved. In this paper, we overcome these challenges by proposing a novel framework called Meta-learning update via Multi-scale Knowledge Distillation and Data Augmentation (MMKDDA). Specifically, we apply multiscale knowledge distillation to grasp the evolution of long-range and short-range spatial relationships at different feature levels to alleviate the problem of data imbalance. Besides, our method mixes the samples from the episodic memory and current task in the online continual training procedure, thus alleviating the side influence due to the change of probability distribution. Moreover, we optimize our model via the meta-learning update resorting to the number of tasks seen previously, which is helpful to keep a better balance between stability and plasticity. Finally, our experimental evaluation on four benchmark datasets shows the effectiveness of the proposed MMKDDA framework against other popular baselines, and ablation studies are also conducted to further analyze the role of each component in our framework.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 48 of its 70 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

48 of 70 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2015Fast R-CNNcited by 2,930
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  9. 2016Learning without Forgettingcited by 2,465
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  12. 2016Progressive Neural Networkscited by 1,332
36 more
  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  2. 2017Continual Learning with Deep Generative Replaycited by 1,152
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  5. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  6. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  7. 2019Large Scale Incremental Learningcited by 727
  8. 2014Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitioncited by 719
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  10. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  11. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  12. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  13. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  14. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  15. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  16. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  17. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  18. 2018Meta-Learning: A Surveycited by 254
  19. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 181
  20. 2020PLOP: Learning without Forgetting for Continual Semantic Segmentationcited by 177
  21. 2019Class-incremental Learning via Deep Model Consolidationcited by 172
  22. 2020Adversarial Continual Learningcited by 140
  23. 2018Continuous Learning in Single-Incremental-Task Scenarioscited by 109
  24. 2020iTAML: An Incremental Task-Agnostic Meta-learning Approachcited by 90
  25. 2018Fully Convolutional Adaptation Networks for Semantic Segmentationcited by 75
  26. 2020La-MAML: Look-ahead Meta Learning for Continual Learningcited by 51
  27. 2018Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learningcited by 34
  28. 2021Self-Supervised Video Representation Learning with Meta-Contrastive Networkcited by 28
  29. 2019M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learningcited by 21
  30. 2019Efficient Continual Learning in Neural Networks with Embedding Regularizationcited by 20
  31. 2020Adaptive feature recombination and recalibration for semantic segmentation with Fully Convolutional Networkscited by 13
  32. 2019Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigencecited by 12
  33. 2020Bilevel Continual Learningcited by 10
  34. 2017Inception Recurrent Convolutional Neural Network for Object Recognitioncited by 5
  35. 2019Reducing Catastrophic Forgetting in Modular Neural Networks by Dynamic Information Balancingcited by 4
  36. 2020Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memorycited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 2.

6 papers.

  1. 2023Rethinking Momentum Knowledge Distillation in Online Continual Learningcited by 18
  2. 2024Unveiling the Anomalies in an Ever-Changing World: A Benchmark for Pixel-Level Anomaly Detection in Continual Learningcited by 13
  3. 2022Continual Learning Approaches for Anomaly Detectioncited by 9
  4. 2023Online Continual Learning via the Knowledge Invariant and Spread-out Propertiescited by 1
  5. 2025Learning Dynamic Representations via An Optimally-Weighted Maximum Mean Discrepancy Optimization Framework for Continual Learningcited by 0
  6. 2025Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCILcited by 0