mlsanityData to 7 Oct 2026

When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective

arXiv 2409.01821 ↗First posted 3 Sep 2024cs.CV cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Adapting pre-trained models to new tasks can exhibit varying effectiveness across datasets. Visual prompting, a state-of-the-art parameter-efficient transfer learning method, can significantly improve the performance of out-of-distribution tasks. On the other hand, linear probing, a standard transfer learning method, can sometimes become the best approach. We propose a log-likelihood ratio (LLR) approach to analyze the comparative benefits of visual prompting and linear probing. By employing the LLR score alongside resource-efficient visual prompts approximations, our cost-effective measure attains up to a 100-fold reduction in run time compared to full training, while achieving prediction accuracies up to 91%. The source code is available at https://github.com/IBM/VP-LLR.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 45 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

29 of 45 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  11. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  12. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
17 more
  1. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  2. 2017Functional Map of the Worldcited by 338
  3. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  4. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
  5. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  6. 2019Transferability and Hardness of Supervised Classification Taskscited by 125
  7. 2018Adversarial Reprogramming of Neural Networkscited by 115
  8. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  9. 2020Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resourcescited by 83
  10. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  11. 2020Understanding Adversarial Examples from the Mutual Influence of Images and Perturbationscited by 59
  12. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  13. 2023AutoVP: An Automated Visual Prompting Framework and Benchmarkcited by 29
  14. 2023ETran: Energy-Based Transferability Estimationcited by 18
  15. 2022Neural Clamping: Joint Input Perturbation and Temperature Scaling for Neural Network Calibrationcited by 4
  16. 2022NCTV: Neural Clamping Toolkit and Visualization for Neural Network Calibrationcited by 2
  17. 2023Fourier Sensitivity and Regularization of Computer Vision Modelscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domainscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  2. 2023LaViP:Language-Grounded Visual Prompts
  3. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  4. 2024Attention Prompting on Image for Large Vision-Language Models
  5. 2025Bayesian Principles Improve Prompt Learning In Vision-Language Models
  6. 2025Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
  7. 2023Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models
  8. 2022Unified Vision and Language Prompt Learning