Towards Optimal Adapter Placement for Efficient Transfer Learning
Not on the map. Why?
Abstract (arXiv)
Parameter-efficient transfer learning (PETL) aims to adapt pre-trained models to new downstream tasks while minimizing the number of fine-tuned parameters. Adapters, a popular approach in PETL, inject additional capacity into existing networks by incorporating low-rank projections, achieving performance comparable to full fine-tuning with significantly fewer parameters. This paper investigates the relationship between the placement of an adapter and its performance. We observe that adapter location within a network significantly impacts its effectiveness, and that the optimal placement is task-dependent. To exploit this observation, we introduce an extended search space of adapter connections, including long-range and recurrent adapters. We demonstrate that even randomly selected adapter placements from this expanded space yield improved results, and that high-performing placements often correlate with high gradient rank. Our findings reveal that a small number of strategically placed adapters can match or exceed the performance of the common baseline of adding adapters in every block, opening a new avenue for research into optimal adapter placement strategies.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 56 of its 65 references that are in the map. First posted on arXiv 21 Oct 2024.
Builds on
56 of 65 references.
References · 56
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2022Visual Prompt Tuningcited by 1,681
- 2021Scaling Vision Transformerscited by 999
38 more
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
- 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
- 2017Learning multiple visual domains with residual adapterscited by 557
- 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
- 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
- 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
- 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
- 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
- 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
- 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
- 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
- 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
- 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
- 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
- 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
- 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
- 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
- 2022Neural Prompt Searchcited by 135
- 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
- 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
- 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
- 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 121
- 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
- 2024Higher Layers Need More LoRA Expertscited by 102
- 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
- 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
- 2022GradMax: Growing Neural Networks using Gradient Informationcited by 66
- 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
- 2022Rank Diminishing in Deep Neural Networkscited by 47
- 2019The intriguing role of module criticality in the generalization of deep networkscited by 40
- 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 35
- 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.