mlsanityData to 14 Oct 2026

SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes

arXiv 2506.20990 ↗First posted 26 Jun 2025cs.LG cs.CL cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Fine-tuning vision language models (VLMs) has achieved remarkable performance across various downstream tasks; yet, it requires access to model gradients through backpropagation (BP), making them unsuitable for memory-constrained, inference-only edge devices. To address this limitation, previous work has explored various BP-free fine-tuning methods. However, these approaches often rely on high-variance evolutionary strategies (ES) or zeroth-order (ZO) optimization, and often fail to achieve satisfactory performance. In this paper, we propose a hybrid Sharpness-aware Zeroth-order optimization (SharpZO) approach, specifically designed to enhance the performance of ZO VLM fine-tuning via a sharpness-aware warm-up training. SharpZO features a two-stage optimization process: a sharpness-aware ES stage that globally explores and smooths the loss landscape to construct a strong initialization, followed by a fine-grained local search via sparse ZO optimization. The entire optimization relies solely on forward passes. Detailed theoretical analysis and extensive experiments on CLIP models demonstrate that SharpZO significantly improves accuracy and convergence speed, achieving up to 7% average gain over state-of-the-art forward-only methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 56 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

6 of 56 references.

References · 6

  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  2. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  3. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
  4. 2025QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Modelscited by 11
  5. 2024HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimizationcited by 9
  6. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  2. 2026Flatness Preserves Instruction Following in Vision-Language-Action Modelscited by 1
  3. 2026GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuningcited by 0
  4. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Gradient constrained sharpness-aware prompt learning for vision-language models
  2. 2026ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
  3. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models
  4. 2025MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models
  5. 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
  6. 2025ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models
  7. 2025GLAD: Generalizable Tuning for Vision-Language Models
  8. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning