# LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation

arXiv: https://arxiv.org/abs/2502.00896 · HTML: https://mlsanity.com/paper/2502.00896/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 2 Feb 2025.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data](https://mlsanity.com/paper/1808.01204/) (2018)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Understanding and Improving Visual Prompting: A Label-Mapping Perspective](https://mlsanity.com/paper/2211.11635/) (2022)
- [Improving Tuning-Free Real Image Editing with Proximal Guidance](https://mlsanity.com/paper/2306.05414/) (2023)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [On the Convergence Rate of Training Recurrent Neural Networks](https://mlsanity.com/paper/1810.12065/) (2018)
- [BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning](https://mlsanity.com/paper/2303.14773/) (2023)
- [Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian](https://mlsanity.com/paper/1906.05392/) (2019)
- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)

## Built on by (most cited first, 17 of 17)

- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [MTL-UE: Learning to Learn Nothing for Multi-Task Learning](https://mlsanity.com/paper/2505.05279/) (2025)
- [LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation](https://mlsanity.com/paper/2503.13794/) (2025)
- [Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts](https://mlsanity.com/paper/2506.01000/) (2025)
- [Multi-Scale Visual Prompting for Lightweight Small-Image Classification](https://mlsanity.com/paper/2512.03663/) (2025)
- [Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models](https://mlsanity.com/paper/2606.26379/) (2026)
- [Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models](https://mlsanity.com/paper/2603.21426/) (2026)
- [AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition](https://mlsanity.com/paper/2508.11870/) (2025)
- [Meow: End-to-End Outline Writing for Automatic Academic Survey](https://mlsanity.com/paper/2509.19370/) (2025)
- [Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation](https://mlsanity.com/paper/2510.07823/) (2025)
- [Unified Graph Prompt Learning via Low-Rank Graph Message Prompting](https://mlsanity.com/paper/2604.11257/) (2026)
- [Text-Guided Multi-Scale Frequency Representation Adaptation](https://mlsanity.com/paper/2605.08181/) (2026)
- [Beyond Low-Rank: Low-Rank Sparse Prompting via Spiking Neural Network and Prompt Factorization](https://mlsanity.com/paper/2606.01945/) (2026)
- [Adapting Vision Foundation Models with Cascaded Semantics](https://mlsanity.com/paper/2608.05393/) (2026)
- [ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation](https://mlsanity.com/paper/2608.21194/) (2026)
- [You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming](https://mlsanity.com/paper/2609.36661/) (2026)
- [Reprogramming Vision-Language Models via Structured Prompt Reparameterization](https://mlsanity.com/paper/2609.36680/) (2026)

## Abstract (arXiv)

Visual prompting has gained popularity as a method for adapting pre-trained models to specific tasks, particularly in the realm of parameter-efficient tuning. However, existing visual prompting techniques often pad the prompt parameters around the image, limiting the interaction between the visual prompts and the original image to a small set of patches while neglecting the inductive bias present in shared information across different patches. In this study, we conduct a thorough preliminary investigation to identify and address these limitations. We propose a novel visual prompt design, introducing Low-Rank matrix multiplication for Visual Prompting (LoR-VP), which enables shared and patch-specific information across rows and columns of image pixels. Extensive experiments across seven network architectures and four datasets demonstrate significant improvements in both performance and efficiency compared to state-of-the-art visual prompting methods, achieving up to 6 times faster training times, utilizing 18 times fewer visual prompt parameters, and delivering a 3.1% improvement in performance. The code is available as https://github.com/jincan333/LoR-VP.