mlsanityData to 7 Oct 2026

A Comprehensive Analysis of Adapter Efficiency

arXiv 2305.07491 ↗First posted 12 May 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Adapters have been positioned as a parameter-efficient fine-tuning (PEFT) approach, whereby a minimal number of parameters are added to the model and fine-tuned. However, adapters have not been sufficiently analyzed to understand if PEFT translates to benefits in training/deployment efficiency and maintainability/extensibility. Through extensive experiments on many adapters, tasks, and languages in supervised and cross-lingual zero-shot settings, we clearly show that for Natural Language Understanding (NLU) tasks, the parameter efficiency in adapters does not translate to efficiency gains compared to full fine-tuning of models. More precisely, adapters are relatively expensive to train and have slightly higher deployment latency. Furthermore, the maintainability/extensibility benefits of adapters can be achieved with simpler approaches like multi-task training via full fine-tuning, which also provide relatively faster training times. We, therefore, recommend that for moderately sized models for NLU tasks, practitioners should rely on full fine-tuning or multi-task training rather than using adapters. Our code is available at https://github.com/AI4Bharat/adapter-efficiency.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 12 May 2023.

Builds on

35 of 54 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
23 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  5. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  9. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  10. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  11. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  12. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  13. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  14. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  15. 2020The Multilingual Amazon Reviews Corpuscited by 142
  16. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  17. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  18. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  19. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  20. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 57
  21. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  22. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  23. 2021SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensionscited by 6

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 3.

8 papers.

  1. 2024Large Language Models Meet NLP: A Surveycited by 102
  2. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  3. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  4. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  5. 2024Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restorationcited by 4
  6. 2025Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languagescited by 3
  7. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  8. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2