mlsanityData to 14 Oct 2026

AdapterDrop: On the Efficiency of Adapters in Transformers

arXiv 2010.11918 ↗First posted 22 Oct 2020cs.LG cs.CLcited by 224

Not on the map. Why?

Abstract (arXiv)

Massively pre-trained transformer models are computationally expensive to fine-tune, slow for inference, and have large storage requirements. Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters. In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions. We show that AdapterDrop can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances. We further prune adapters from AdapterFusion, which improves the inference efficiency while maintaining the task performances entirely.

Key facts. Cited by 224 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 40; 2023: 53; 2024: 64; 2025: 40; 2026: 17.

224 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  4. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  5. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  6. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  7. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  8. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  9. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  10. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  11. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  12. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
38 more
  1. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  2. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  3. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  4. 2023Modular Deep Learningcited by 83
  5. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  6. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  7. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  8. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 71
  9. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  10. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  11. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  12. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 63
  13. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  14. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  15. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  16. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  17. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  18. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 48
  19. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  20. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  21. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  22. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  23. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  24. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  25. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 35
  26. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  27. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  28. 2022AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Taskscited by 29
  29. 2023Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuningcited by 29
  30. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 29
  31. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  32. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  33. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  34. 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 25
  35. 2023Parameter-efficient Modularised Bias Mitigation via AdapterFusioncited by 25
  36. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  37. 2023Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictionscited by 24
  38. 2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24

50 most cited of 224 All 224 (JSON)