# Make Continual Learning Stronger via C-Flat

arXiv: https://arxiv.org/abs/2404.00986 · HTML: https://mlsanity.com/paper/2404.00986/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 74 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 30 of 57)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Surrogate Gap Minimization Improves Sharpness-Aware Training](https://mlsanity.com/paper/2203.08065/) (2022)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [Adaptive Aggregation Networks for Class-Incremental Learning](https://mlsanity.com/paper/2010.05063/) (2020)
- [Towards Understanding Sharpness-Aware Minimization](https://mlsanity.com/paper/2206.06232/) (2022)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)
- [Scalable and Order-robust Continual Learning with Additive Parameter Decomposition](https://mlsanity.com/paper/1902.09432/) (2019)
- [Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima](https://mlsanity.com/paper/2111.01549/) (2021)
- [Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2203.06359/) (2022)
- [Conditional Channel Gated Networks for Task-Aware Continual Learning](https://mlsanity.com/paper/2004.00070/) (2020)

## Built on by (most cited first, 30 of 35)

- [Continual Multimodal Contrastive Learning](https://mlsanity.com/paper/2503.14963/) (2025)
- [Revisiting Neural Networks for Continual Learning: An Architectural Perspective](https://mlsanity.com/paper/2404.14829/) (2024)
- [Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective](https://mlsanity.com/paper/2506.03951/) (2025)
- [MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems](https://mlsanity.com/paper/2503.16549/) (2025)
- [Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting](https://mlsanity.com/paper/2605.02105/) (2026)
- [ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think](https://mlsanity.com/paper/2501.01045/) (2025)
- [When Continue Learning Meets Multimodal Large Language Model: A Survey](https://mlsanity.com/paper/2503.01887/) (2025)
- [Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2503.05423/) (2025)
- [Continual GUI Agents](https://mlsanity.com/paper/2601.20732/) (2026)
- [Sequence Transferability and Task Order Selection in Continual Learning](https://mlsanity.com/paper/2502.06544/) (2025)
- [Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning](https://mlsanity.com/paper/2605.05676/) (2026)
- [Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2506.12409/) (2025)
- [Lessons and Open Questions from a Unified Study of Camera-Trap Species Recognition Over Time](https://mlsanity.com/paper/2603.20509/) (2026)
- [Do Neural Networks Lose Plasticity in a Gradually Changing World?](https://mlsanity.com/paper/2602.09234/) (2026)
- [Hyperbolic Multimodal Continual Learning](https://mlsanity.com/paper/2608.09572/) (2026)
- [A Flat Minima Perspective on Understanding Augmentations and Model Robustness](https://mlsanity.com/paper/2505.24592/) (2025)
- [C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning](https://mlsanity.com/paper/2508.18860/) (2025)
- [More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning](https://mlsanity.com/paper/2510.21019/) (2025)
- [A Faster Path to Continual Learning](https://mlsanity.com/paper/2604.11064/) (2026)
- [5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2606.10488/) (2026)
- [Dual Prototypes for Adaptive Pre-Trained Model in Class-Incremental Learning](https://mlsanity.com/paper/2411.17766/) (2024)
- [First-place Solution for Streetscape Shop Sign Recognition Competition](https://mlsanity.com/paper/2501.02811/) (2025)
- [Towards Robust Incremental Learning under Ambiguous Supervision](https://mlsanity.com/paper/2501.13584/) (2025)
- [Ken Utilization Layer: Hebbian Replay Within a Student's Ken for Adaptive Exercise Recommendation](https://mlsanity.com/paper/2507.00032/) (2025)
- [Empirical Evidences for the Effects of Feature Diversity in Open Set Recognition and Continual Learning](https://mlsanity.com/paper/2508.13005/) (2025)
- [EndoCIL: A Class-Incremental Learning Framework for Endoscopic Image Classification](https://mlsanity.com/paper/2510.17200/) (2025)
- [Pushing the Limits of Distillation-Based Continual Learning via Classifier-Proximal Lightweight Plugins](https://mlsanity.com/paper/2512.03537/) (2025)
- [Beyond Sharpness: A Flatness Decomposition Framework for Efficient Continual Learning](https://mlsanity.com/paper/2601.07636/) (2026)
- [Finding Structure in Continual Learning](https://mlsanity.com/paper/2602.04555/) (2026)
- [Memory-efficient Continual Learning with Prototypical Exemplar Condensation](https://mlsanity.com/paper/2603.13804/) (2026)

## Abstract (arXiv)

Model generalization ability upon incrementally acquiring dynamically updating knowledge from sequentially arriving tasks is crucial to tackle the sensitivity-stability dilemma in Continual Learning (CL). Weight loss landscape sharpness minimization seeking for flat minima lying in neighborhoods with uniform low loss or smooth gradient is proven to be a strong training regime improving model generalization compared with loss minimization based optimizer like SGD. Yet only a few works have discussed this training regime for CL, proving that dedicated designed zeroth-order sharpness optimizer can improve CL performance. In this work, we propose a Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for CL. C-Flat could be easily called with only one line of code and is plug-and-play to any CL methods. A general framework of C-Flat applied to all CL categories and a thorough comparison with loss minima optimizer and flat minima based CL approaches is presented in this paper, showing that our method can boost CL performance in almost all cases. Code is available at https://github.com/WanNaa/C-Flat.