# Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing

arXiv: https://arxiv.org/abs/2310.06234 · HTML: https://mlsanity.com/paper/2310.06234/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 42 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)

## Built on by (most cited first, 30 of 35)

- [Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying](https://mlsanity.com/paper/2311.09578/) (2023)
- [Visual Fourier Prompt Tuning](https://mlsanity.com/paper/2411.01327/) (2024)
- [Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark](https://mlsanity.com/paper/2402.02242/) (2024)
- [Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach](https://mlsanity.com/paper/2403.19067/) (2024)
- [PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization](https://mlsanity.com/paper/2409.17137/) (2024)
- [Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters](https://mlsanity.com/paper/2506.14530/) (2025)
- [Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformers](https://mlsanity.com/paper/2312.15681/) (2023)
- [Selective Visual Prompting in Vision Mamba](https://mlsanity.com/paper/2412.08947/) (2024)
- [SAMIHS: Adaptation of Segment Anything Model for Intracranial Hemorrhage Segmentation](https://mlsanity.com/paper/2311.08190/) (2023)
- [Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters](https://mlsanity.com/paper/2402.00828/) (2024)
- [Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation](https://mlsanity.com/paper/2410.22952/) (2024)
- [Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction](https://mlsanity.com/paper/2411.01158/) (2024)
- [Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2506.04453/) (2025)
- [Efficient Adaptation of Pre-trained Vision Transformer underpinned by Approximately Orthogonal Fine-Tuning Strategy](https://mlsanity.com/paper/2507.13260/) (2025)
- [iDAT: inverse Distillation Adapter-Tuning](https://mlsanity.com/paper/2403.15750/) (2024)
- [Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer](https://mlsanity.com/paper/2409.13999/) (2024)
- [PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation](https://mlsanity.com/paper/2506.02453/) (2025)
- [Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models](https://mlsanity.com/paper/2606.26379/) (2026)
- [Mixture of Physical Priors Adapter for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2412.02759/) (2024)
- [AdaptSR: Low-Rank Adaptation for Efficient and Scalable Real-World Super-Resolution](https://mlsanity.com/paper/2503.07748/) (2025)
- [APLA: A Simple Adaptation Method for Vision Transformers](https://mlsanity.com/paper/2503.11335/) (2025)
- [MedSeqFT: Sequential Fine-tuning Foundation Models for 3D Medical Image Segmentation](https://mlsanity.com/paper/2509.06096/) (2025)
- [A Multi-dimensional Framework for Evaluating Generalization in EEG Foundation Models](https://mlsanity.com/paper/2605.28563/) (2026)
- [Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning](https://mlsanity.com/paper/2501.06884/) (2025)
- [LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMs](https://mlsanity.com/paper/2507.01806/) (2025)
- [Regularizing Subspace Redundancy of Low-Rank Adaptation](https://mlsanity.com/paper/2507.20745/) (2025)
- [Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformers](https://mlsanity.com/paper/2512.24603/) (2025)
- [A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters](https://mlsanity.com/paper/2603.11211/) (2026)
- [From Pen to Pixel: Translating Hand-Drawn Plots into Graphical APIs via a Novel Benchmark and Efficient Adapter](https://mlsanity.com/paper/2603.26356/) (2026)
- [LoCO: Low-rank Compositional Rotation Fine-tuning](https://mlsanity.com/paper/2605.15916/) (2026)

## Abstract (arXiv)

The advent of high-capacity pre-trained models has revolutionized problem-solving in computer vision, shifting the focus from training task-specific models to adapting pre-trained models. Consequently, effectively adapting large pre-trained models to downstream tasks in an efficient manner has become a prominent research area. Existing solutions primarily concentrate on designing lightweight adapters and their interaction with pre-trained models, with the goal of minimizing the number of parameters requiring updates. In this study, we propose a novel Adapter Re-Composing (ARC) strategy that addresses efficient pre-trained model adaptation from a fresh perspective. Our approach considers the reusability of adaptation parameters and introduces a parameter-sharing scheme. Specifically, we leverage symmetric down-/up-projections to construct bottleneck operations, which are shared across layers. By learning low-dimensional re-scaling coefficients, we can effectively re-compose layer-adaptive adapters. This parameter-sharing strategy in adapter design allows us to significantly reduce the number of new parameters while maintaining satisfactory performance, thereby offering a promising approach to compress the adaptation cost. We conduct experiments on 24 downstream image classification tasks using various Vision Transformer variants to evaluate our method. The results demonstrate that our approach achieves compelling transfer learning performance with a reduced parameter count. Our code is available at hrefhttps://github.com/DavidYanAnDe/ARChttps://github.com/DavidYanAnDe/ARC.