mlsanityData to 7 Oct 2026

Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification

arXiv 2409.16718 ↗First posted 25 Sep 2024cs.CV cs.AI cs.CLcited by 14

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances in fine-tuning Vision-Language Models (VLMs) have witnessed the success of prompt tuning and adapter tuning, while the classic model fine-tuning on inherent parameters seems to be overlooked. It is believed that fine-tuning the parameters of VLMs with few-shot samples corrupts the pre-trained knowledge since fine-tuning the CLIP model even degrades performance. In this paper, we revisit this viewpoint, and propose a new perspective: fine-tuning the specific parameters instead of all will uncover the power of classic model fine-tuning on VLMs. Through our meticulous study, we propose ClipFit, a simple yet effective method to fine-tune CLIP without introducing any overhead of extra parameters. We demonstrate that by only fine-tuning the specific bias terms and normalization layers, ClipFit can improve the performance of zero-shot CLIP by 7.27% average harmonic mean accuracy. Lastly, to understand how fine-tuning in CLIPFit affects the pre-trained models, we conducted extensive experimental analyses w.r.t. changes in internal parameters and representations. We found that low-level text bias layers and the first layer normalization layer change much more than other layers. The code is available at urlhttps://github.com/minglllli/CLIPFit.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 52 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

34 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  2. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  3. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct

Other references · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2016Layer Normalizationcited by 5,579
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
18 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  6. 2021Learning to Prompt for Continual Learningcited by 938
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  8. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  10. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  11. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  12. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  13. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  14. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  15. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  16. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  17. 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
  18. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 3.

14 papers.

Sort:

By area:

  1. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  2. 2025Deepfake Detection that Generalizes Across Benchmarkscited by 20
  3. 2024EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmarkcited by 14
  4. 2025Unlocking the Hidden Potential of CLIP in Generalizable Deepfake Detectioncited by 10
  5. 2025Approximate Domain Unlearning for Vision-Language Modelscited by 6
  6. 2025VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Modelscited by 2
  7. 2025Patient-Specific Autoregressive Models for Organ Motion Prediction in Radiotherapycited by 1
  8. 2025Improving Generalization of Language-Conditioned Robot Manipulationcited by 1
  9. 2025Spotlighter: Revisiting Prompt Tuning from a Representative Mining Viewcited by 1
  10. 2025Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filteringcited by 1
  11. 2025Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Modelscited by 0
  12. 2026When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learningcited by 0
2 more
  1. 2026A3B2: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learningcited by 0
  2. 2026Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
  2. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values
  3. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  4. 2022Localized Latent Updates for Fine-Tuning Vision-Language Models
  5. 2024Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization
  6. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts
  7. 2024Towards Compatible Fine-tuning for Vision-Language Model Updates
  8. 2024Beyond Accuracy: On the Effects of Fine-tuning Towards Vision-Language Model's Prediction Rationality