mlsanityData to 7 Oct 2026

AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

arXiv 2210.17451 ↗First posted 31 Oct 2022cs.CL cs.AI cs.LGcited by 137

Not on the map. Why?

Cited by 32 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules -- given the underlying PEFT method of choice -- introduced in each Transformer layer while keeping most of the PLM weights frozen. For instance, AdaMix can leverage a mixture of adapters like Houlsby or a mixture of low rank decomposition matrices like LoRA to improve downstream task performance over the corresponding PEFT methods for fully supervised and few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the same computational cost and the number of tunable parameters as the underlying PEFT method. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

38 of 49 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
26 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2017Deep Bayesian Active Learning with Image Datacited by 812
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  12. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  13. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  14. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  15. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  16. 2020What is being transferred in transfer learning?cited by 365
  17. 2021True Few-Shot Learning with Language Modelscited by 343
  18. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  19. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  20. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  21. 2021Hash Layers For Large Sparse Modelscited by 236
  22. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  23. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  24. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  25. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  26. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 59; 2025: 34; 2026: 19.

137 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  5. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  6. 2023Efficient Large Language Models: A Surveycited by 183
  7. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  8. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  9. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  10. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  11. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  12. 2023Modular Deep Learningcited by 82
38 more
  1. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  2. 2024Mixture of A Million Expertscited by 70
  3. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  4. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  5. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  6. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  7. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  8. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  9. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  10. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  11. 2023OWL: A Large Language Model for IT Operationscited by 43
  12. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  13. 2024LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Taskscited by 39
  14. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  15. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  16. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  17. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  18. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  19. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  20. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoEcited by 25
  21. 2024BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptationcited by 25
  22. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  23. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  24. 2024QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptationcited by 23
  25. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22
  26. 2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18
  27. 2024Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgettingcited by 17
  28. 2025Improved Representation Steering for Language Modelscited by 17
  29. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  30. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  31. 2024LLM-based multi-agent poetry generation in non-cooperative environmentscited by 15
  32. 2025Mixture of Experts in Large Language Modelscited by 15
  33. 2024AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guaranteescited by 14
  34. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  35. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  36. 2023DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rulescited by 12
  37. 2024Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Modelscited by 11
  38. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10

50 most cited of 137 All 137 (JSON)