mlsanityData to 14 Oct 2026

Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach

arXiv 2403.19067 ↗First posted 28 Mar 2024cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient fine-tuning for pre-trained Vision Transformers aims to adeptly tailor a model to downstream tasks by learning a minimal set of new adaptation parameters while preserving the frozen majority of pre-trained parameters. Striking a balance between retaining the generalizable representation capacity of the pre-trained model and acquiring task-specific features poses a key challenge. Currently, there is a lack of focus on guiding this delicate trade-off. In this study, we approach the problem from the perspective of Singular Value Decomposition (SVD) of pre-trained parameter matrices, providing insights into the tuning dynamics of existing methods. Building upon this understanding, we propose a Residual-based Low-Rank Rescaling (RLRR) fine-tuning strategy. This strategy not only enhances flexibility in parameter tuning but also ensures that new parameters do not deviate excessively from the pre-trained model through a residual design. Extensive experiments demonstrate that our method achieves competitive performance across various downstream image classification tasks, all while maintaining comparable new parameters. We believe this work takes a step forward in offering a unified perspective for interpreting existing methods and serves as motivation for the development of new approaches that move closer to effectively considering the crucial trade-off mentioned above. Our code is available at hrefhttps://github.com/zstarN70/RLRR.githttps://github.com/zstarN70/RLRR.git.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

24 of 32 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  12. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
12 more
  1. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  3. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  4. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  5. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  6. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  7. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  8. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  9. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  10. 2017Fine-Grained Car Detection for Visual Census Estimationcited by 76
  11. 2022A Review of Deep Transfer Learning and Recent Advancementscited by 57
  12. 2023Efficient Adaptation of Large Vision Transformer via Adapter Re-Composingcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 4.

18 papers.

Sort:

By area:

  1. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  2. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  3. 2024Efficient Training of Large Vision Models via Advanced Automated Progressive Learningcited by 5
  4. 2024Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformationcited by 5
  5. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  6. 2025Efficient Adaptation of Pre-trained Vision Transformer underpinned by Approximately Orthogonal Fine-Tuning Strategycited by 3
  7. 2024Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformercited by 2
  8. 2025DFQ-ViT: Data-Free Quantization for Vision Transformers without Fine-tuningcited by 2
  9. 2024Mixture of Physical Priors Adapter for Parameter-Efficient Fine-Tuningcited by 1
  10. 2025APLA: A Simple Adaptation Method for Vision Transformerscited by 1
  11. 2026PriorNet: Prior-Guided Engagement Estimation from Face Videocited by 1
  12. 2024Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learningcited by 0
6 more
  1. 2025Regularizing Subspace Redundancy of Low-Rank Adaptationcited by 0
  2. 2025Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Modelscited by 0
  3. 2025Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformerscited by 0
  4. 2026Alignment-Aware Model Adaptation via Feedback-Guided Optimizationcited by 0
  5. 2026Early Intervention for VFM-based Multimodal Medical Image Classificationcited by 0
  6. 2026MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformerscited by 0