mlsanityData to 7 Oct 2026

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think

arXiv 2501.01045 ↗First posted 2 Jan 2025cs.CV cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Backpropagation provides a generalized configuration for overcoming catastrophic forgetting. Optimizers such as SGD and Adam are commonly used for weight updates in continual learning and continual pre-training. However, access to gradient information is not always feasible in practice due to black-box APIs, hardware constraints, or non-differentiable systems, a challenge we refer to as the gradient bans. To bridge this gap, we introduce ZeroFlow, the first benchmark designed to evaluate gradient-free optimization algorithms for overcoming forgetting. ZeroFlow examines a suite of forward pass-based methods across various algorithms, forgetting scenarios, and datasets. Our results show that forward passes alone can be sufficient to mitigate forgetting. We uncover novel optimization principles that highlight the potential of forward pass-based methods in mitigating forgetting, managing task conflicts, and reducing memory demands. Additionally, we propose new enhancements that further improve forgetting resistance using only forward passes. This work provides essential tools and insights to advance the development of forward-pass-based methods for continual learning.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 58 of its 78 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

58 of 78 references.

References · 58

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  4. 2017Continual Learning with Deep Generative Replaycited by 1,152
  5. 2018Experience Replay for Continual Learningcited by 1,074
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  7. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  8. 2019On the Convergence of Adam and Beyondcited by 834
  9. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  10. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  11. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  12. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
38 more
  1. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  2. 2021Gradient Projection Memory for Continual Learningcited by 338
  3. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  4. 2018Deep Learning in Spiking Neural Networkscited by 258
  5. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  6. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  7. 2023Class-Incremental Learning: A Surveycited by 209
  8. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  9. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  10. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  11. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  12. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  13. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  14. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  15. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  16. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  17. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
  18. 2023Equivariant Multi-Modality Image Fusioncited by 98
  19. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  20. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  21. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  22. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 75
  23. 2022Gradients without Backpropagationcited by 73
  24. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  25. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
  26. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 66
  27. 2022Scaling Forward Gradient With Local Lossescited by 65
  28. 2023PILOT: A Pre-Trained Model-Based Continual Learning Toolboxcited by 64
  29. 2020CPR: Classifier-Projection Regularization for Continual Learningcited by 62
  30. 2022ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protectioncited by 56
  31. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  32. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  33. 2016A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Ordercited by 43
  34. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  35. 2024Image Fusion via Vision-Language Modelcited by 40
  36. 2021MEGEX: Data-Free Model Extraction Attack against Gradient-Based Explainable AIcited by 35
  37. 2024Make Continual Learning Stronger via C-Flatcited by 35
  38. 2023BiRT: Bio-inspired Replay in Vision Transformers for Continual Learningcited by 25

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspectivecited by 14
  2. 2026Continual GUI Agentscited by 7
  3. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Modelscited by 3
  4. 2026Lessons and Open Questions from a Unified Study of Camera-Trap Species Recognition Over Timecited by 2
  5. 2025C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learningcited by 1
  6. 2025More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learningcited by 1
  7. 2026A Faster Path to Continual Learningcited by 1
  8. 2026Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theorycited by 0
  9. 2026Continual Learning in Transitioncited by 0