# When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective

arXiv: https://arxiv.org/abs/2409.01821 · HTML: https://mlsanity.com/paper/2409.01821/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [Transferability and Hardness of Supervised Classification Tasks](https://mlsanity.com/paper/1908.08142/) (2019)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Understanding and Improving Visual Prompting: A Label-Mapping Perspective](https://mlsanity.com/paper/2211.11635/) (2022)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning](https://mlsanity.com/paper/2202.10629/) (2022)
- [Understanding Adversarial Examples from the Mutual Influence of Images and Perturbations](https://mlsanity.com/paper/2007.06189/) (2020)
- [PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks](https://mlsanity.com/paper/2203.05126/) (2022)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)
- [ETran: Energy-Based Transferability Estimation](https://mlsanity.com/paper/2308.02027/) (2023)
- [Neural Clamping: Joint Input Perturbation and Temperature Scaling for Neural Network Calibration](https://mlsanity.com/paper/2209.11604/) (2022)
- [NCTV: Neural Clamping Toolkit and Visualization for Neural Network Calibration](https://mlsanity.com/paper/2211.16274/) (2022)
- [Fourier Sensitivity and Regularization of Computer Vision Models](https://mlsanity.com/paper/2301.13514/) (2023)

## Built on by (most cited first, 1 of 1)

- [Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains](https://mlsanity.com/paper/2512.22664/) (2025)

## Abstract (arXiv)

Adapting pre-trained models to new tasks can exhibit varying effectiveness across datasets. Visual prompting, a state-of-the-art parameter-efficient transfer learning method, can significantly improve the performance of out-of-distribution tasks. On the other hand, linear probing, a standard transfer learning method, can sometimes become the best approach. We propose a log-likelihood ratio (LLR) approach to analyze the comparative benefits of visual prompting and linear probing. By employing the LLR score alongside resource-efficient visual prompts approximations, our cost-effective measure attains up to a 100-fold reduction in run time compared to full training, while achieving prediction accuracies up to 91%. The source code is available at https://github.com/IBM/VP-LLR.