mlsanityData to 7 Oct 2026

AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

arXiv 2205.12410 ↗First posted 24 May 2022cs.CL cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules -- given the underlying PEFT method of choice -- introduced in each Transformer layer while keeping most of the PLM weights frozen. For instance, AdaMix can leverage a mixture of adapters like Houlsby or a mixture of low rank decomposition matrices like LoRA to improve downstream task performance over the corresponding PEFT methods for fully supervised and few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the same computational cost and the number of tunable parameters as the underlying PEFT method. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

38 of 49 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
26 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2017Deep Bayesian Active Learning with Image Datacited by 812
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  12. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  13. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  14. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  15. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  16. 2020What is being transferred in transfer learning?cited by 365
  17. 2021True Few-Shot Learning with Language Modelscited by 343
  18. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  19. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  20. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  21. 2021Hash Layers For Large Sparse Modelscited by 236
  22. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  23. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  24. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  25. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  26. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 2.

7 papers.

  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  2. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
  3. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  4. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Surveycited by 22
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  6. 2024SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptationcited by 3
  7. 2025Rethinking Data: Towards Better Performing Domain-Specific Small Language Modelscited by 0