mlsanityData to 7 Oct 2026

Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?

arXiv 2401.12902 ↗First posted 23 Jan 2024cs.CVcited by 31

Not on the map. Why?

Abstract (arXiv)

As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficient transfer learning technique has gained attention due to its superior performance compared to traditional full-finetuning. However, the conditions favoring VPT (the ``when") and the underlying rationale (the ``why") remain unclear. In this paper, we conduct a comprehensive analysis across 19 distinct datasets and tasks. To understand the ``when" aspect, we identify the scenarios where VPT proves favorable by two dimensions: task objectives and data distributions. We find that VPT is preferrable when there is 1) a substantial disparity between the original and the downstream task objectives (e.g., transitioning from classification to counting), or 2) a similarity in data distributions between the two tasks (e.g., both involve natural images). In exploring the ``why" dimension, our results indicate VPT's success cannot be attributed solely to overfitting and optimization considerations. The unique way VPT preserves original features and adds parameters appears to be a pivotal factor. Our study provides insights into VPT's mechanisms, and offers guidance for its optimal utilization.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 94 of its 133 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

94 of 133 references.

References · 94

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
38 more
  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  11. 2016Matching Networks for One Shot Learningcited by 2,246
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  13. 2019Searching for MobileNetV3cited by 1,984
  14. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  15. 2022Visual Prompt Tuningcited by 1,674
  16. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  17. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  18. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  19. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  20. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  21. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  22. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  24. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  25. 2021Going deeper with Image Transformerscited by 759
  26. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  27. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  28. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  29. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  30. 2021ImageNet-21K Pretraining for the Massescited by 607
  31. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  32. 2017Learning multiple visual domains with residual adapterscited by 553
  33. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  34. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  35. 2019Res2Net: A New Multi-scale Backbone Architecturecited by 468
  36. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  37. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  38. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 15; 2026: 7.

31 papers.

Sort:

By area:

  1. 2024Visual Fourier Prompt Tuningcited by 30
  2. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  3. 2024Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learningcited by 19
  4. 2024M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learningcited by 16
  5. 2024Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognitioncited by 15
  6. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  7. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Modelscited by 12
  8. 2024Unsupervised Domain Adaption Harnessing Vision-Language Pre-trainingcited by 12
  9. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  10. 2025PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocationcited by 9
  11. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  12. 2025Efficient Multi-modal Long Context Learning for Training-free Adaptationcited by 7
19 more
  1. 2026Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptationcited by 7
  2. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
  3. 2025MEPT: Mixture of Expert Prompt Tuning as a Manifold Mappercited by 5
  4. 2024Prototypical Transformer as Unified Motion Learnerscited by 4
  5. 2025Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuningcited by 4
  6. 2024MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motionscited by 3
  7. 2025eSkinHealth: A Multimodal Dataset for Neglected Tropical Skin Diseasescited by 3
  8. 2025Self-Supervised Visual Prompting for Cross-Domain Road Damage Detectioncited by 3
  9. 2024Leveraging Foundation Models for Efficient Federated Learning in Resource-restricted Edge Networkscited by 2
  10. 2026Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Modelscited by 1
  11. 2025Adept: Annotation-Denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretrainingcited by 0
  12. 2025A Foundation Model for DAS Signal Recognition and Visual Prompt Tuning of the Pre-trained Model for Downstream Taskscited by 0
  13. 2025Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigationcited by 0
  14. 2025Structured Output Regularization: a framework for few-shot transfer learningcited by 0
  15. 2026TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditchingcited by 0
  16. 2026Inheritance Between Feedforward and Convolutional Networks via Model Projectioncited by 0
  17. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Modelscited by 0
  18. 2026Adapting Vision Foundation Models with Cascaded Semanticscited by 0
  19. 2026Decomposition-Guided Diffusion Language Models for Inertial Confinement Fusion Predictioncited by 0