mlsanityData to 14 Oct 2026

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

arXiv 2309.05444 ↗First posted 11 Sep 2023cs.CL cs.LGcited by 150

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

The Mixture of Experts (MoE) is a widely known neural architecture where an ensemble of specialized sub-models optimizes overall performance with a constant computational cost. However, conventional MoEs pose challenges at scale due to the need to store all experts in memory. In this paper, we push MoE to the limit. We propose extremely parameter-efficient MoE by uniquely combining MoE architecture with lightweight experts.Our MoE architecture outperforms standard parameter-efficient fine-tuning (PEFT) methods and is on par with full fine-tuning by only updating the lightweight experts -- less than 1% of an 11B parameters model. Furthermore, our method generalizes to unseen tasks as it does not depend on any prior task knowledge. Our research underscores the versatility of the mixture of experts architecture, showcasing its ability to deliver robust performance even when subjected to rigorous parameter constraints. Our code used in all the experiments is publicly available here: https://github.com/for-ai/parameter-efficient-moe.

Key facts. Cited by 150 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 57 of its 64 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

57 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning7 Jun

Other references · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  13. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  14. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  15. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  16. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  17. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  18. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  19. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  20. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  21. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  22. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  23. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  24. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  25. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  26. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  27. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  28. 2021Hash Layers For Large Sparse Modelscited by 237
  29. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  30. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  31. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  32. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  33. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  34. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  35. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  36. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  37. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  38. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 61; 2025: 43; 2026: 37.

150 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  4. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  5. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 111
  6. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  7. 2024Higher Layers Need More LoRA Expertscited by 102
  8. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  9. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  10. 2024Mixture of A Million Expertscited by 72
  11. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  12. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
38 more
  1. 2024Theory on Mixture-of-Experts in Continual Learningcited by 54
  2. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  3. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  4. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
  5. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  6. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 44
  7. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  8. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  9. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  10. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 34
  11. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  12. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 31
  13. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  14. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 23
  15. 2023Label Supervised LLaMA Finetuningcited by 20
  16. 2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoEcited by 19
  17. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  18. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 18
  19. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 17
  20. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  21. 2024Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptationcited by 15
  22. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 15
  23. 2025Mixture of Experts in Large Language Modelscited by 15
  24. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  25. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  26. 2025PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learningcited by 14
  27. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 13
  28. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 13
  29. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 13
  30. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  31. 2024Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligencecited by 11
  32. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  33. 2024Conditional computation in neural networks: principles and research trendscited by 10
  34. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  35. 2024Low-Rank Interconnected Adaptation across Layerscited by 10
  36. 2024Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Fieldcited by 10
  37. 2024MEMoE: Enhancing Model Editing with Mixture of Experts Adaptorscited by 9
  38. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9

50 most cited of 150 All 150 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts
  2. 2023Improving Expert Specialization in Mixture of Experts
  3. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
  4. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  5. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  6. 2026ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
  7. 2026MoE Lens -- An Expert Is All You Need
  8. 2024A Closer Look into Mixture-of-Experts in Large Language Models