mlsanityData to 14 Oct 2026

Make Continual Learning Stronger via C-Flat

arXiv 2404.00986 ↗First posted 1 Apr 2024cs.LG cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Model generalization ability upon incrementally acquiring dynamically updating knowledge from sequentially arriving tasks is crucial to tackle the sensitivity-stability dilemma in Continual Learning (CL). Weight loss landscape sharpness minimization seeking for flat minima lying in neighborhoods with uniform low loss or smooth gradient is proven to be a strong training regime improving model generalization compared with loss minimization based optimizer like SGD. Yet only a few works have discussed this training regime for CL, proving that dedicated designed zeroth-order sharpness optimizer can improve CL performance. In this work, we propose a Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for CL. C-Flat could be easily called with only one line of code and is plug-and-play to any CL methods. A general framework of C-Flat applied to all CL categories and a thorough comparison with loss minima optimizer and flat minima based CL approaches is presented in this paper, showing that our method can boost CL performance in almost all cases. Code is available at https://github.com/WanNaa/C-Flat.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

57 of 74 references.

References · 57

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2016Learning without Forgettingcited by 2,496
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2018Experience Replay for Continual Learningcited by 1,095
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  10. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  11. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  12. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
38 more
  1. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  2. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  3. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  4. 2021Gradient Projection Memory for Continual Learningcited by 348
  5. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  6. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  7. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 229
  8. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  9. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  10. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 154
  11. 2020Adaptive Aggregation Networks for Class-Incremental Learningcited by 149
  12. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  13. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  14. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  15. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  16. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  17. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  18. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  19. 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 100
  20. 2022Sharpness-Aware Training for Freecited by 95
  21. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  22. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 86
  23. 2021PyCIL: A Python Toolbox for Class-Incremental Learningcited by 83
  24. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 82
  25. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 72
  26. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  27. 2020CPR: Classifier-Projection Regularization for Continual Learningcited by 62
  28. 2020Gradient-based Editing of Memory Examples for Online Task-free Continual Learningcited by 62
  29. 2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
  30. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  31. 2021Subspace Regularizers for Few-Shot Class Incremental Learningcited by 56
  32. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 55
  33. 2023PCR: Proxy-based Contrastive Replay for Online Class-Incremental Continual Learningcited by 52
  34. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  35. 2023Dense Network Expansion for Class Incremental Learningcited by 45
  36. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 45
  37. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  38. 2023Parameter-Level Soft-Masking for Continual Learningcited by 40

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 16.

35 papers.

Sort:

By area:

  1. 2025Continual Multimodal Contrastive Learningcited by 20
  2. 2024Revisiting Neural Networks for Continual Learning: An Architectural Perspectivecited by 17
  3. 2025Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspectivecited by 14
  4. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problemscited by 10
  5. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 10
  6. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  7. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  8. 2025Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental Learningcited by 7
  9. 2026Continual GUI Agentscited by 7
  10. 2025Sequence Transferability and Task Order Selection in Continual Learningcited by 5
  11. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  12. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Modelscited by 3
23 more
  1. 2026Lessons and Open Questions from a Unified Study of Camera-Trap Species Recognition Over Timecited by 3
  2. 2026Do Neural Networks Lose Plasticity in a Gradually Changing World?cited by 2
  3. 2026Hyperbolic Multimodal Continual Learningcited by 2
  4. 2025A Flat Minima Perspective on Understanding Augmentations and Model Robustnesscited by 1
  5. 2025C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learningcited by 1
  6. 2025More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learningcited by 1
  7. 2026A Faster Path to Continual Learningcited by 1
  8. 20265% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuningcited by 1
  9. 2024Dual Prototypes for Adaptive Pre-Trained Model in Class-Incremental Learningcited by 0
  10. 2025First-place Solution for Streetscape Shop Sign Recognition Competitioncited by 0
  11. 2025Towards Robust Incremental Learning under Ambiguous Supervisioncited by 0
  12. 2025Ken Utilization Layer: Hebbian Replay Within a Student's Ken for Adaptive Exercise Recommendationcited by 0
  13. 2025Empirical Evidences for the Effects of Feature Diversity in Open Set Recognition and Continual Learningcited by 0
  14. 2025EndoCIL: A Class-Incremental Learning Framework for Endoscopic Image Classificationcited by 0
  15. 2025Pushing the Limits of Distillation-Based Continual Learning via Classifier-Proximal Lightweight Pluginscited by 0
  16. 2026Beyond Sharpness: A Flatness Decomposition Framework for Efficient Continual Learningcited by 0
  17. 2026Finding Structure in Continual Learningcited by 0
  18. 2026Memory-efficient Continual Learning with Prototypical Exemplar Condensationcited by 0
  19. 2026Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theorycited by 0
  20. 2026Lost or Hidden? A Concept-Level Forgetting in Supervised Continual Learningcited by 0
  21. 2026Continual Learning in Transitioncited by 0
  22. 2026Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solutioncited by 0
  23. 2026Normative Loss Landscape Navigation: A Trajectory-Based Approach to Mitigating Forgetting in Incremental Learningcited by 0