mlsanityData to 7 Oct 2026

Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models

arXiv 2203.01104 ↗First posted 2 Mar 2022cs.CL cs.AI cs.LGcited by 31

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Recently, Mixture-of-Experts (short as MoE) architecture has achieved remarkable success in increasing the model capacity of large-scale language models. However, MoE requires incorporating significantly more parameters than the base model being extended. In this paper, we propose building a parameter-efficient MoE architecture by sharing information among experts. We adopt the matrix product operator (MPO, a tensor decomposition from quantum many-body physics) to reconstruct the parameter matrix in the expert layer and increase model capacity for pre-trained language models by sharing parameters of the central tensor (containing the core information) among different experts while enabling the specificity through the auxiliary tensors (complementing the central tensor) of different experts. To address the unbalanced optimization issue, we further design the gradient mask strategy for the MPO-based MoE architecture. Extensive experiments based on T5 and GPT-2 show improved performance and efficiency of the pre-trained language model (27.2x reduction in total parameters for the superior model performance, compared with the Switch Transformers). Our code is publicly available at https://github.com/RUCAIBox/MPOE.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 29 of its 46 references that are in the map. First posted on arXiv 2 Mar 2022.

Builds on

29 of 46 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  12. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
17 more
  1. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  2. 2015Tensorizing Neural Networkscited by 246
  3. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  4. 2021Hash Layers For Large Sparse Modelscited by 236
  5. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  6. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  7. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  8. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  9. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  10. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  11. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 55
  12. 2020Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Datacited by 50
  13. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 28
  14. 2019Compressing deep neural networks by matrix product operatorscited by 27
  15. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22
  16. 2020Regularizing Meta-Learning via Gradient Dropoutcited by 15
  17. 2020A Model Compression Method with Matrix Product Operators for Speech Enhancementcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 9; 2025: 7; 2026: 5.

31 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  3. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  5. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  6. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  7. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
  8. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  9. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  10. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  11. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  12. 2022PAD-Net: An Efficient Framework for Dynamic Networkscited by 9
19 more
  1. 2025EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Modelscited by 7
  2. 2023MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answeringcited by 5
  3. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  4. 2024MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfercited by 5
  5. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
  6. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  7. 2025Sample Selection via Contrastive Fragmentation for Noisy Label Regressioncited by 4
  8. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacitycited by 3
  9. 2023Condensing Multilingual Knowledge with Lightweight Language-Specific Modulescited by 3
  10. 2026Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance Learningcited by 3
  11. 2023Domain-Agnostic Neural Architecture for Class Incremental Continual Learning in Document Processing Platformcited by 2
  12. 2025Reproducibility Study of "Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents"cited by 2
  13. 2025CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edgecited by 2
  14. 2025Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Makingcited by 2
  15. 2026Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Expertscited by 1
  16. 2023Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecturecited by 0
  17. 2023Subspace-Configurable Networkscited by 0
  18. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptationcited by 0
  19. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefillcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  3. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  4. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
  5. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  6. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading
  7. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  8. 2025Advancing Expert Specialization for Better MoE