mlsanityData to 7 Oct 2026

AdapterFusion: Non-Destructive Task Composition for Transfer Learning

arXiv 2005.00247 ↗First posted 1 May 2020cs.CLcited by 780

Not on the map. Why?

Cited by 27 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Sequential fine-tuning and multi-task learning are methods aiming to incorporate knowledge from multiple tasks; however, they suffer from catastrophic forgetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks. First, in the knowledge extraction stage we learn task specific parameters called adapters, that encapsulate the task-specific information. We then combine the adapters in a separate knowledge composition step. We show that by separating the two stages, i.e., knowledge extraction and knowledge composition, the classifier can effectively exploit the representations learned from multiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it effectively combines various types of knowledge at different layers of the model. We show that our approach outperforms traditional strategies such as full fine-tuning as well as multi-task learning. Our code and adapters are available at AdapterHub.ml.

Key facts. Cited by 780 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 45; 2022: 93; 2023: 155; 2024: 218; 2025: 155; 2026: 114.

780 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2021Learning to Prompt for Continual Learningcited by 938
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  9. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  10. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  11. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  12. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
38 more
  1. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  2. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  3. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  4. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  5. 2023Personalize Segment Anything Model with One Shotcited by 250
  6. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  7. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  8. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  9. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  10. 2023Continual Pre-training of Language Modelscited by 194
  11. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  12. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  13. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  14. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  15. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  16. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  17. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  18. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  19. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  20. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  21. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  22. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  23. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  24. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  25. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  26. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  27. 2022Efficient Few-Shot Learning Without Promptscited by 135
  28. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  29. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  30. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  31. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
  32. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  33. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  34. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 120
  35. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  36. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  37. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  38. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104

50 most cited of 780 All 780 (JSON)