# Understanding and Improving Visual Prompting: A Label-Mapping Perspective

arXiv: https://arxiv.org/abs/2211.11635 · HTML: https://mlsanity.com/paper/2211.11635/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 60 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [Unified Vision and Language Prompt Learning](https://mlsanity.com/paper/2210.07225/) (2022)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Visual Prompt Tuning for Test-time Domain Adaptation](https://mlsanity.com/paper/2210.04831/) (2022)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [Visual Prompt Tuning for Generative Transfer Learning](https://mlsanity.com/paper/2210.00990/) (2022)
- [Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning](https://mlsanity.com/paper/2202.10629/) (2022)
- [Domain Adaptive Neural Networks for Object Recognition](https://mlsanity.com/paper/1409.6041/) (2014)
- [Unadversarial Examples: Designing Objects for Robust Vision](https://mlsanity.com/paper/2012.12235/) (2020)
- [How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspective](https://mlsanity.com/paper/2203.14195/) (2022)
- [Cross-modal Adversarial Reprogramming](https://mlsanity.com/paper/2102.07325/) (2021)
- [Visual Prompting for Adversarial Robustness](https://mlsanity.com/paper/2210.06284/) (2022)
- [Fairness Reprogramming](https://mlsanity.com/paper/2209.10222/) (2022)

## Built on by (most cited first, 30 of 87)

- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [All in One and One for All: A Simple yet Effective Method towards Cross-domain Graph Pretraining](https://mlsanity.com/paper/2402.09834/) (2024)
- [ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphs](https://mlsanity.com/paper/2402.11235/) (2024)
- [Visual Prompting in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2409.15310/) (2024)
- [Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies](https://mlsanity.com/paper/2410.19878/) (2024)
- [Parameter-Efficient Fine-Tuning for Foundation Models](https://mlsanity.com/paper/2501.13787/) (2025)
- [An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learning](https://mlsanity.com/paper/2308.00788/) (2023)
- [Unleashing the Power of Visual Prompting At the Pixel Level](https://mlsanity.com/paper/2212.10556/) (2022)
- [Instruct Me More! Random Prompting for Visual In-Context Learning](https://mlsanity.com/paper/2311.03648/) (2023)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)
- [Unifying Image Processing as Visual Prompting Question Answering](https://mlsanity.com/paper/2310.10513/) (2023)
- [CoLLaVO: Crayon Large Language and Vision mOdel](https://mlsanity.com/paper/2402.11248/) (2024)
- [Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective](https://mlsanity.com/paper/2312.01397/) (2023)
- [Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark](https://mlsanity.com/paper/2402.02242/) (2024)
- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://mlsanity.com/paper/2303.04995/) (2023)
- [Sample-specific Masks for Visual Reprogramming-based Prompting](https://mlsanity.com/paper/2406.03150/) (2024)
- [Exploring the Benefits of Visual Prompting in Differential Privacy](https://mlsanity.com/paper/2303.12247/) (2023)
- [LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation](https://mlsanity.com/paper/2502.00896/) (2025)
- [Bayesian-guided Label Mapping for Visual Reprogramming](https://mlsanity.com/paper/2410.24018/) (2024)
- [Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning](https://mlsanity.com/paper/2310.08782/) (2023)
- [Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models](https://mlsanity.com/paper/2405.01468/) (2024)
- [PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocation](https://mlsanity.com/paper/2503.06901/) (2025)
- [Quantifying Privacy Risks of Prompts in Visual Prompt Learning](https://mlsanity.com/paper/2310.11970/) (2023)
- [HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization](https://mlsanity.com/paper/2401.09716/) (2024)
- [Attribute-based Visual Reprogramming for Vision-Language Models](https://mlsanity.com/paper/2501.13982/) (2025)
- [MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learning](https://mlsanity.com/paper/2307.05707/) (2023)
- [Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification](https://mlsanity.com/paper/2412.16780/) (2024)
- [DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models](https://mlsanity.com/paper/2505.24025/) (2025)

## Abstract (arXiv)

We revisit and advance visual prompting (VP), an input prompting technique for vision tasks. VP can reprogram a fixed, pre-trained source model to accomplish downstream tasks in the target domain by simply incorporating universal prompts (in terms of input perturbation patterns) into downstream data points. Yet, it remains elusive why VP stays effective even given a ruleless label mapping (LM) between the source classes and the target classes. Inspired by the above, we ask: How is LM interrelated with VP? And how to exploit such a relationship to improve its accuracy on target tasks? We peer into the influence of LM on VP and provide an affirmative answer that a better 'quality' of LM (assessed by mapping precision and explanation) can consistently improve the effectiveness of VP. This is in contrast to the prior art where the factor of LM was missing. To optimize LM, we propose a new VP framework, termed ILM-VP (iterative label mapping-based visual prompting), which automatically re-maps the source labels to the target labels and progressively improves the target task accuracy of VP. Further, when using a contrastive language-image pretrained (CLIP) model, we propose to integrate an LM process to assist the text prompt selection of CLIP and to improve the target task accuracy. Extensive experiments demonstrate that our proposal significantly outperforms state-of-the-art VP methods. As highlighted below, we show that when reprogramming an ImageNet-pretrained ResNet-18 to 13 target tasks, our method outperforms baselines by a substantial margin, e.g., 7.9% and 6.7% accuracy improvements in transfer learning to the target Flowers102 and CIFAR100 datasets. Besides, our proposal on CLIP-based VP provides 13.7% and 7.1% accuracy improvements on Flowers102 and DTD respectively. Our code is available at https://github.com/OPTML-Group/ILM-VP.