mlsanityData to 14 Oct 2026

Do We Really Need a Large Number of Visual Prompts?

arXiv 2305.17223 ↗First posted 26 May 2023cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Due to increasing interest in adapting models on resource-constrained edges, parameter-efficient transfer learning has been widely explored. Among various methods, Visual Prompt Tuning (VPT), prepending learnable prompts to input space, shows competitive fine-tuning performance compared to training of full network parameters. However, VPT increases the number of input tokens, resulting in additional computational overhead. In this paper, we analyze the impact of the number of prompts on fine-tuning performance and self-attention operation in a vision transformer architecture. Through theoretical and empirical analysis we show that adding more prompts does not lead to linear performance improvement. Further, we propose a Prompt Condensation (PC) technique that aims to prevent performance degradation from using a small number of prompts. We validate our methods on FGVC and VTAB-1k tasks and show that our approach reduces the number of prompts by ~70% while maintaining accuracy.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

36 of 49 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  7. 2022Visual Prompt Tuningcited by 1,681
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  9. 2016Progressive Neural Networkscited by 1,361
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  11. 2022Token Merging: Your ViT But Fastercited by 1,021
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
24 more
  1. 2020Efficient Transformers: A Surveycited by 790
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  3. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  4. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  5. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  6. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  7. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  8. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  9. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  10. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  11. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  12. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  13. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  14. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  15. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  16. 2021Learned Token Pruning for Transformerscited by 139
  17. 2022Neural Prompt Searchcited by 135
  18. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 121
  19. 2017Fine-Grained Car Detection for Visual Census Estimationcited by 76
  20. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  21. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  22. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  23. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  24. 2019Backprop with Approximate Activations for Memory-efficient Network Trainingcited by 19

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 5.

9 papers.

  1. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  2. 2025PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocationcited by 9
  3. 2024MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusioncited by 5
  4. 2025Token Coordinated Prompt Attention is Needed for Visual Promptingcited by 3
  5. 2024Proactive Schemes: A Survey of Adversarial Attacks for Social Goodcited by 2
  6. 2025Multi-Scale Visual Prompting for Lightweight Small-Image Classificationcited by 2
  7. 2024Harnessing Vision-Language Pretrained Models with Temporal-Aware Adaptation for Referring Video Object Segmentationcited by 1
  8. 2025VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansioncited by 1
  9. 2023Conditional Prompt Tuning for Multimodal Fusioncited by 0