mlsanityData to 7 Oct 2026

Exploring Visual Prompts for Adapting Large-Scale Models

arXiv 2203.17274 ↗First posted 31 Mar 2022cs.CVcited by 271

Not on the map. Why?

Cited by 70 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We investigate the efficacy of visual prompting to adapt large-scale models in vision. Following the recent approach from prompt tuning and adversarial reprogramming, we learn a single image perturbation such that a frozen model prompted with this perturbation performs a new task. Through comprehensive experiments, we demonstrate that visual prompting is particularly effective for CLIP and robust to distribution shift, achieving performance competitive with standard linear probes. We further analyze properties of the downstream dataset, prompt design, and output transformation in regard to adaptation performance. The surprising effectiveness of visual prompting provides a new perspective on adapting pre-trained models in vision. Code is available at http://hjbahng.github.io/visual_prompting .

Key facts. Cited by 271 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

35 of 51 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2022Visual Prompt Tuningcited by 1,674
23 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,079
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  7. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  9. 2021GPT Understands, Toocited by 715
  10. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  11. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  13. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  14. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  15. 2017Functional Map of the Worldcited by 338
  16. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  17. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  18. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  19. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  20. 2018Adversarial Reprogramming of Neural Networkscited by 115
  21. 2020The iWildCam 2020 Competition Datasetcited by 86
  22. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  23. 2020Unadversarial Examples: Designing Objects for Robust Visioncited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 81; 2024: 80; 2025: 62; 2026: 28.

271 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  3. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  4. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  5. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  6. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  7. 2023Visual Prompt Multi-Modal Trackingcited by 141
  8. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  9. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  10. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  11. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  12. 2022Universal Prompt Tuning for Graph Neural Networkscited by 100
38 more
  1. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  2. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalizationcited by 100
  3. 2022Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptationcited by 98
  4. 2022Visual Prompt Tuning for Test-time Domain Adaptationcited by 97
  5. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  6. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  7. 2023Fine-Grained Visual Promptingcited by 81
  8. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  9. 2024Test-Time Model Adaptation with Only Forward Passescited by 74
  10. 2023Consistency-guided Prompt Learning for Vision-Language Modelscited by 73
  11. 2022LPT: Long-tailed Prompt Tuning for Image Classificationcited by 69
  12. 2024Sparse autoencoders reveal selective remapping of visual concepts during adaptationcited by 66
  13. 2022Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNetscited by 62
  14. 2023Segment Any Anomaly without Training via Hybrid Prompt Regularizationcited by 59
  15. 2023ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptationcited by 58
  16. 2023Diversity-Aware Meta Visual Promptingcited by 57
  17. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  18. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 51
  19. 2023AutoAD: Movie Description in Contextcited by 50
  20. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  21. 2023Towards In-context Scene Understandingcited by 45
  22. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  23. 2023Dual-path Adaptation from Image to Video Transformerscited by 40
  24. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  25. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  26. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 39
  27. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  28. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?cited by 38
  29. 2023SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matchingcited by 38
  30. 2022Unleashing the Power of Visual Prompting At the Pixel Levelcited by 37
  31. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 36
  32. 2023Progressive Visual Prompt Learning with Contrastive Feature Re-formationcited by 35
  33. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  34. 2023Affective Faces for Goal-Driven Dyadic Communicationcited by 34
  35. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  36. 2023EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understandingcited by 33
  37. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  38. 2024Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptationcited by 32

50 most cited of 271 All 271 (JSON)