mlsanityData to 7 Oct 2026

PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocation

arXiv 2503.06901 ↗First posted 10 Mar 2025cs.CV cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Visual prompt tuning (VPT), i.e., fine-tuning some lightweight prompt tokens, provides an efficient and effective approach for adapting pre-trained models to various downstream tasks. However, most prior art indiscriminately uses a fixed prompt distribution across different tasks, neglecting the importance of each block varying depending on the task. In this paper, we introduce adaptive distribution optimization (ADO) by tackling two key questions: (1) How to appropriately and formally define ADO, and (2) How to design an adaptive distribution strategy guided by this definition? Through empirical analysis, we first confirm that properly adjusting the distribution significantly improves VPT performance, and further uncover a key insight that a nested relationship exists between ADO and VPT. Based on these findings, we propose a new VPT framework, termed PRO-VPT (iterative Prompt RelOcation-based VPT), which adaptively adjusts the distribution built upon a nested optimization formulation. Specifically, we develop a prompt relocation strategy derived from this formulation, comprising two steps: pruning idle prompts from prompt-saturated blocks, followed by allocating these prompts to the most prompt-needed blocks. By iteratively performing prompt relocation and VPT, our proposal can adaptively learn the optimal prompt distribution in a nested optimization-based manner, thereby unlocking the full potential of VPT. Extensive experiments demonstrate that our proposal significantly outperforms advanced VPT methods, e.g., PRO-VPT surpasses VPT by 1.6 pp and 2.0 pp average accuracy, leading prompt-based methods to state-of-the-art performance on VTAB-1k and FGVC benchmarks. The code is available at https://github.com/ckshang/PRO-VPT.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 54 of its 70 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

54 of 70 references.

References · 54

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Mask R-CNNcited by 7,259
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2022Visual Prompt Tuningcited by 1,674
38 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  2. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  3. 2021Transformers in Vision: A Surveycited by 1,009
  4. 2023Vision Transformers Need Registerscited by 844
  5. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  6. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  7. 2017Deep Reinforcement Learning: An Overviewcited by 327
  8. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  9. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  10. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  11. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  12. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 166
  13. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  14. 2022Neural Prompt Searchcited by 133
  15. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  16. 2019Winning the Lottery with Continuous Sparsificationcited by 99
  17. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  18. 2019DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networkscited by 80
  19. 2017Fine-Grained Car Detection for Visual Census Estimationcited by 76
  20. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  21. 2021Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learningcited by 72
  22. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  23. 2020Distance-Based Regularisation of Deep Networks for Fine-Tuningcited by 47
  24. 2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
  25. 2023Progressive Visual Prompt Learning with Contrastive Feature Re-formationcited by 35
  26. 2023Visual Tuningcited by 34
  27. 2024Revisiting the Power of Prompt for Visual Tuningcited by 32
  28. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  29. 2022XPrompt: Exploring the Extreme of Prompt Tuningcited by 30
  30. 2023DTL: Disentangled Transfer Learning for Visual Recognitioncited by 22
  31. 2023Learning Expressive Prompting With Residuals for Vision Transformerscited by 21
  32. 2023SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channelscited by 20
  33. 2023Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptationcited by 18
  34. 2023Fast Trainable Projection for Robust Fine-Tuningcited by 18
  35. 2024Adapters Strike Backcited by 16
  36. 2024Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognitioncited by 15
  37. 2023Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptationcited by 14
  38. 2023Less is More: Selective Layer Finetuning with SubTuningcited by 13

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learningcited by 1
  2. 2026Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Modelscited by 1
  3. 2026How to Ask the AI: A User Perspective Survey for Large Language Model Promptingcited by 1
  4. 2026Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compressioncited by 0
  5. 2026Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lassocited by 0
  6. 2026Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Datacited by 0
  7. 2026Exposing Functional Fusion: A New Class of Strategic Backdoor in Dynamic Prompt Architecturescited by 0
  8. 2026Circuit Fine-Tuning for Compute-Efficient Transformer Adaptationcited by 0
  9. 2026Text-Guided Visual Dependency Graph Learning with Cross-Modal Attention Priorscited by 0