mlsanityData to 7 Oct 2026

AutoVP: An Automated Visual Prompting Framework and Benchmark

arXiv 2310.08381 ↗First posted 12 Oct 2023cs.CV cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Visual prompting (VP) is an emerging parameter-efficient fine-tuning approach to adapting pre-trained vision models to solve various downstream image-classification tasks. However, there has hitherto been little systematic study of the design space of VP and no clear benchmark for evaluating its performance. To bridge this gap, we propose AutoVP, an end-to-end expandable framework for automating VP design choices, along with 12 downstream image-classification tasks that can serve as a holistic VP-performance benchmark. Our design space covers 1) the joint optimization of the prompts; 2) the selection of pre-trained models, including image classifiers and text-image encoders; and 3) model output mapping strategies, including nonparametric and trainable label mapping. Our extensive experimental results show that AutoVP outperforms the best-known current VP methods by a substantial margin, having up to 6.7% improvement in accuracy; and attains a maximum performance increase of 27.5% compared to linear-probing (LP) baseline. AutoVP thus makes a two-fold contribution: serving both as an efficient tool for hyperparameter tuning on VP design choices, and as a comprehensive benchmark that can reasonably be expected to accelerate VP's development. The source code is available at https://github.com/IBM/AutoVP.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

43 of 65 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  12. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
31 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2022Visual Prompt Tuningcited by 1,674
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  8. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  9. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  10. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  11. 2017Functional Map of the Worldcited by 338
  12. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  13. 2022Visual Prompting via Image Inpaintingcited by 248
  14. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
  15. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  16. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  17. 2018Adversarial Reprogramming of Neural Networkscited by 115
  18. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  19. 2020Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resourcescited by 83
  20. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  21. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  22. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  23. 2022Unleashing the Power of Visual Prompting At the Pixel Levelcited by 37
  24. 2022Visual Prompting for Adversarial Robustnesscited by 33
  25. 2023Convolutional Visual Prompt for Robust Visual Perceptioncited by 17
  26. 2023A Three-regime Model of Network Pruningcited by 15
  27. 2023Prompt Space Optimizing Few-shot Reasoning Success with Large Language Modelscited by 11
  28. 2022Neural Clamping: Joint Input Perturbation and Temperature Scaling for Neural Network Calibrationcited by 4
  29. 2023Rethinking Visual Prompt Learning as Masked Visual Token Modelingcited by 4
  30. 2023From Visual Prompt Learning to Zero-Shot Transfer: Mapping Is All You Needcited by 4
  31. 2022NCTV: Neural Clamping Toolkit and Visualization for Neural Network Calibrationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 12; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  2. 2024Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detectioncited by 30
  3. 2025LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptationcited by 17
  4. 2024Sample-specific Masks for Visual Reprogramming-based Promptingcited by 16
  5. 2024GDA: Generalized Diffusion for Robust Test-time Adaptationcited by 14
  6. 2024Bayesian-guided Label Mapping for Visual Reprogrammingcited by 12
  7. 2024Subgraph-level Universal Prompt Tuningcited by 9
  8. 2025Attribute-based Visual Reprogramming for Vision-Language Modelscited by 7
  9. 2024Selective Visual Prompting in Vision Mambacited by 6
  10. 2025DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Modelscited by 5
  11. 2025Understanding Model Reprogramming for CLIP via Decoupling Visual Promptscited by 4
  12. 2024Robust Adaptation of Foundation Models with Black-Box Visual Promptingcited by 3
17 more
  1. 2025Token Coordinated Prompt Attention is Needed for Visual Promptingcited by 3
  2. 2025Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instructioncited by 3
  3. 2024Proactive Schemes: A Survey of Adversarial Attacks for Social Goodcited by 2
  4. 2025Model Reprogramming Demystified: A Neural Tangent Kernel Perspectivecited by 2
  5. 2025State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understandingcited by 2
  6. 2024Training Spatial-Frequency Visual Prompts and Probabilistic Clusters for Accurate Black-Box Transfer Learningcited by 1
  7. 2024When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspectivecited by 1
  8. 2024Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuningcited by 1
  9. 2025E-InMeMo: Enhanced Prompting for Visual In-Context Learningcited by 1
  10. 2025Towards Single-Source Domain Generalized Object Detection via Causal Visual Promptscited by 1
  11. 2025Parameter-Efficient Adaptation of mPLUG-Owl2 via Pixel-Level Visual Prompts for NR-IQAcited by 0
  12. 2025Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigationcited by 0
  13. 2026Beyond Low-Rank: Low-Rank Sparse Prompting via Spiking Neural Network and Prompt Factorizationcited by 0
  14. 2026BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogrammingcited by 0
  15. 2026ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptationcited by 0
  16. 2026You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogrammingcited by 0
  17. 2026Reprogramming Vision-Language Models via Structured Prompt Reparameterizationcited by 0