mlsanityData to 14 Oct 2026

Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models

arXiv 2506.12409 ↗First posted 14 Jun 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-Language Continual Learning (VLCL) has attracted significant research attention for its robust capabilities, and the adoption of Parameter-Efficient Fine-Tuning (PEFT) strategies is enabling these models to achieve competitive performance with substantially reduced resource consumption. However, dominated First-Order (FO) optimization is prone to trap models in suboptimal local minima, especially in limited exploration subspace within PEFT. To overcome this challenge, this paper pioneers a systematic exploration of adopting Zeroth-Order (ZO) optimization for PEFT-based VLCL. We first identify the incompatibility of naive full-ZO adoption in VLCL due to optimization process instability. We then investigate the application of ZO optimization from a modality branch-wise to a fine-grained layer-wise across various training units to identify an optimal strategy. Besides, a key theoretical insight reveals that vision modality exhibit higher variance than language counterparts in VLCL during the ZO optimization process, and we propose a modality-aware ZO strategy, which adopts gradient sign normalization in ZO and constrains vision modality perturbation to further improve performance. Benefiting from the adoption of ZO optimization, PEFT-based VLCL fulfills better ability to escape local minima during the optimization process, extensive experiments on four benchmarks demonstrate that our method achieves state-of-the-art results.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 51 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

41 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  2. 2023Learning without Forgetting for Vision-Language Models30 May
  3. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2016Learning without Forgettingcited by 2,496
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  7. 2016An overview of gradient descent optimization algorithmscited by 1,033
  8. 2021Learning to Prompt for Continual Learningcited by 944
  9. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  11. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  12. 2021Gradient Projection Memory for Continual Learningcited by 348
26 more
  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  2. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  3. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  4. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  5. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  6. 2022Sharpness-Aware Training for Freecited by 95
  7. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 82
  8. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 72
  9. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  10. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 68
  11. 2022CLIP model is an Efficient Continual Learnercited by 65
  12. 2020CPR: Classifier-Projection Regularization for Continual Learningcited by 62
  13. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  14. 2024Consistent Prompting for Rehearsal-Free Continual Learningcited by 54
  15. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  16. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47
  17. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  18. 2022SAM as an Optimal Relaxation of Bayescited by 41
  19. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  20. 2024Make Continual Learning Stronger via C-Flatcited by 35
  21. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 19
  22. 2025Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspectivecited by 14
  23. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  24. 2025Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Modelscited by 8
  25. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 4
  26. 2022Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergencecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  2. 2026Harness Continual Learning: Continual Adaptation Beyond Model Parameterscited by 6
  3. 2025C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes
  2. 2026Black-Box Continual Learning for Vision-Language Models
  3. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  4. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion
  5. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting
  6. 2026VLA-ZO: Fast Zeroth-Order Adaptation for Vision-Language-Action Models
  7. 2025ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models
  8. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models